As a Senior Data Engineer I worked with Data Scientists using Amazon SageMaker to analyze data and prototype models. I managed data cataloging and metadata with AWS Glue, and used AWS Redshift for data warehousing and complex SQL queries. I created interactive dashboards with Amazon QuickSight and handled real-time data streaming with AWS Kinesis. I also set up AWS Lambda to automate model retraining and collaborated with AWS DevOps for scalable and secure solutions using CloudFormation and IAM. I deployed web applications using AWS Elastic Beanstalk, built serverless data pipelines with Step Functions and AWS Batch, and integrated machine learning models into AWS IoT Core for predictive maintenance. Additionally, I developed ETL jobs with AWS Glue to clean and map vendor data, storing it in S3 for querying with AWS Athena.

As a Senior Data Engineer I developed and executed system test strategies for data transfer from our internal warehouse to an external entity, automated regular MySQL to Azure SQL Data Warehouse transfers, and led the integration of 142 million records into Azure SQL DW. I converted a local app to Azure-based, redesigned architecture for seamless data integration, and implemented complex Azure Data platform solutions. I utilized Kafka and Spark Streaming for real-time data ingestion, created ETL pipelines in Azure Data Factory, and deployed applications with Terraform. Collaborating with a BI Developer, I set up data pipelines from Azure to Power BI for insightful reporting, focusing on financial and operational data. With strong knowledge of Agile methodologies, I ensured efficient data flow and reporting within the FinOPs domain.

As a Data Engineer, I designed and implemented ETL pipelines to centralize sales data using Python and PostgreSQL, enhancing accuracy and efficiency. Performance tuning on MySQL improved query response times by 30% and reduced resource consumption. Additionally, I created a SQL Server data warehouse for reliable data retrieval and wrote complex SQL queries for reporting purposes. Implementing data validation and error handling in ETL pipelines reduced inconsistencies by 20%. Python scripts automated data tasks, cutting manual efforts by 25%. I also expanded data sources by scraping and ingesting from external APIs. Collaborating with the data science team, I integrated machine learning models into production for real-time data processing.

As a Big Data Developer, I installed and configured Hadoop components like MapReduce, HDFS, and Hive, and developed Java MapReduce jobs for data preprocessing. Using Sqoop, I imported/exported data between HDFS/Hive and other systems like Teradata. I managed cluster maintenance, ETL processes, and data analysis tasks, including Hive queries and PIG scripts for user behavior analysis. Additionally, I utilized Cassandra for data retrieval and Oozie for workflow management.