Senior Data Engineer
CurrentLed the end-to-end data pipeline development, including gathering requirements, designing, and developing solutions.Designed and developed data pipelines using Azure services such as Azure Data Factory, Azure Databricks, andAzure Synapse Analytics to ingest, transform, and load data from various sources into data lakes and data warehouses.Developed data models and implemented data warehouses using Azure Synapse Analytics and SQL Server databases. Implemented and configured Hadoop MapReduce, HDFS, and various AWS services like RedShift, EC2, S3, and EMR for efficient Big Data processing.Utilized Sqoop for seamless import/export operations with HDFS, Pig, Hive, and HBase, enhancing data accessibility and integration.Conducted thorough analysis and management of Hadoop log files to ensure smooth operations and promptly troubleshoot potential issues.Coordinated AWS deployments with Terraform and CloudFormation to ensure smooth system integration.Led the charge in transferring a variety of data from different record-keeping systems into AWS, encouraging cooperation with analysts and system architects to meet business goals.Worked on Azure HD Insight, Databricks, Azure Data Lake, Blob, Azure Data Factory ADF, Synapse, SQL DB, SQL DWH, AWS EC2, S3, EMR, Lambda, Glue, Kinesis.Performing data cleaning and applying transformations using Databricks and Spark data analysis.