Big Data Consultant
Current• Developed and implemented of data ingestion, validation, transformation procedures using AWS services such as Lambda, Glue, Kinesis, SNS, SQS, and CloudFormation.• Developed and maintained scalable data pipelines using AWS Data Services, Hadoop and Spark to process gigabytes and terabytes of data daily/weekly/monthly basis.• Implemented data ingestion solutions using AWS Data Services, Kafka and Flume, leading to a 30% reduction in data latency.• Optimized AWS Athena, Hive queries and Spark jobs, resulting in a 40% increase in processing speed.• Automated ETL workflows using AWS Glue, AWS Cloud Formation, Apache Airflow, reducing manual effort by 50%.• Designed and developed ETL processes for data warehousing solutions on AWS Redshift.• Migrated legacy data systems to modern big data platforms, enhancing data accessibility and performance.• Built real-time data processing systems using AWS Kinesis, AWS Glue, etc.• Collaborated with cross-functional teams to gather and analyze requirements, ensuring data solutions met business needs.• Created and maintained data dashboards using Tableau and AWS Quick sight to provide actionable insights for stakeholders.• Collaborated with data scientists and analysts to design and deploy machine learning models, improving predictive analytics accuracy.