As a Data Engineer, I have been instrumental in designing and setting up the Enterprise Data Lake to support various data storage, processing, analytics, and reporting use cases. My role involves utilizing a wide array of AWS services including S3, EC2, Glue, Athena, and RedShift to manage and transform large datasets. I have developed ETL processes, real-time data ingestion pipelines, and automated workflows to ensure efficient data handling and integration. Additionally, I work on performance… Show more As a Data Engineer, I have been instrumental in designing and setting up the Enterprise Data Lake to support various data storage, processing, analytics, and reporting use cases. My role involves utilizing a wide array of AWS services including S3, EC2, Glue, Athena, and RedShift to manage and transform large datasets. I have developed ETL processes, real-time data ingestion pipelines, and automated workflows to ensure efficient data handling and integration. Additionally, I work on performance tuning, data cleaning, and feature engineering to enhance data quality and accessibility for analytics and machine learning purposes.Key Responsibilities:- Designed and set up the Enterprise Data Lake for various data use cases.- Utilized AWS services: S3, EC2, AWS Glue, Athena, RedShift, EMR, SNS, SQS, DMS,- Extracted data from multiple sources and created tables/databases in the Glue Catalog.- Developed AWS Glue crawlers for data in S3 and RDS.- Designed ETL processes to migrate data from S3 and Parquet/Text Files into AWS Redshift.- Used AWS Glue catalog and Athena for SQL query operations.- Wrote PySpark jobs in AWS Glue for data merging and catalog population.- Automated processes using AWS Glue and Lambda.- Transformed and moved large data sets using AWS EMR and DMS.- Implemented a real-time data ingestion pipeline with Apache Kafka.- Developed PySpark jobs for data transformation and enrichment.- Designed and populated Data Warehouse using AWS Redshift.- Performed performance tuning of Spark Applications.- Conducted data cleaning, feature scaling, and engineering using Python, pandas, and numpy.- Developed and maintained data integration programs in Hadoop and RDBMS environments.- Coordinated with ML teams using Python, Keras, TensorFlow, leveraging Hadoop and other Big Data technologies.- Generated daily reports using Tableau. Show less

My responsibilities included creating end-to-end data storage, integration, processing, and visualization solutions, leveraging Azure SQL Data Warehouse, Data Factory, and Databricks. I developed business intelligence solutions, including dashboards and visualizations using Power BI, to provide data insights and support decision-making. Additionally, I migrated data from traditional systems to Azure databases and optimized Spark applications for efficient data processing.Key… Show more My responsibilities included creating end-to-end data storage, integration, processing, and visualization solutions, leveraging Azure SQL Data Warehouse, Data Factory, and Databricks. I developed business intelligence solutions, including dashboards and visualizations using Power BI, to provide data insights and support decision-making. Additionally, I migrated data from traditional systems to Azure databases and optimized Spark applications for efficient data processing.Key Responsibilities:- Designed and implemented end-to-end data solutions on Azure.- Created database solutions using Azure SQL Data Warehouse and Azure SQL.- Architected and implemented BI solutions using Azure Data Platform services.- Developed dashboards and visualizations with Power BI and SSRS.- Pulled data into Power BI from various sources like SQL Server, Excel, Oracle, and SQL Azure.- Developed conceptual solutions and proof-of-concepts.- Implemented ETL and data movement solutions using Azure Data Factory and SSIS.- Recreated application logic in Azure Data Lake, Data Factory, SQL Database, and SQL Data Warehouse.- Migrated data from traditional databases to Azure.- Interacted with Business Analysts, Users, and SMEs to gather requirements.- Designed Logical and Physical Data Models for Staging, DWH, and Data Mart layers.- Estimated cluster size, monitored, and troubleshooted Spark Databricks clusters.- Tuned Spark Applications for optimal performance.- Created data pipelines, flows, and transformations using ADF and PySpark with Databricks.- Created Requirements Documentation for various projects.- Handled Azure Storage, Blob Storage, File Storage, CDN, and load balancers. Show less