Senior Data Engineer
Current- Developed and optimized T-SQL scripts, including stored procedures and validation scripts, to ensure data integrity and performance.- Designed and managed Azure Data Factory (ADF) pipelines to automate ETL processes and migrate data from SQL Server to Azure SQL Database.- Migrated on-premises data to Azure Data Lake Store (ADLS) and integrated various data types using ADF.- Combined structured, semi-structured, and unstructured data (e.g., log files, media) into Azure Blob Storage using ADF pipelines.- Utilized AWS Glue for ETL workflows, custom Python and PySpark scripts, and managed big data processing with AWS EMR, leveraging Hadoop, Spark, and Hive.- Developed data warehouse process models, including data sourcing, loading, transformation, and extraction.- Designed and implemented star and snowflake schema data models for efficient data warehousing solutions.- Created and scheduled stored procedures in the Azure environment to automate data processing tasks- Implemented Power BI semantic models and visualizations, including parameterized reports, charts, graphs, and drill-down reports, to support business data analysis.- Loaded and analyzed data from Azure Synapse Data Warehouse to provide actionable insights.- Deployed and tested code using Visual Studio Team Services (VSTS), ensuring robust CI/CD pipelines.- Automated ETL processes using SSIS and Azure Data Factory, including package creation and modifications for various data operations.- Implemented CI/CD practices with Jenkins, Azure DevOps, and GitLab CI/CD, automating builds, tests, and deployments. - Prepared source-to-target and data mapping documents to establish relationships and transformations between source and target systems. - Collaborated with subject matter experts to understand business requirements and create effective data migration strategies. - Proficient in using Snowflake for data warehousing, including performance optimization and ETL pipeline development.