Associate Consultant - Data Engineering
Noida, Uttar Pradesh, India
Led migration efforts from MS SQL Server to Snowflake, transferring SQL objects like tables, views, functions and stored proceduresDeveloped JSON mapper files to configure AWS DMS, ensuring seamless data transformationDesigned and implemented complex functions and stored procedures on Snowflake platform to optimise data processing and analysisConducted comprehensive unit testing on all SQL Objects to ensure their proper functionality and integrityUtilised AWS Glue and PySpark to create Data Validation jobs with the following key features: - Checked column count between source and target tables - Verified row count to detect any discrepancies in data transfer - Identified and resolved missing columns between source and target tables - Identified and resolved mismatched data types of columns between source and target tables - Implemented value-by-value comparison to ensure data accuracy and integrity, displaying arrays of non-matching or missing records from the sourceDeveloped Python Scripts for ETL processes, extracting data from MS SQL Server and occasionally CSV files, and transforming it before loading it into Azure Synapse for analysisTools Used: PySpark, Python, AWS Glue, AWS S3, AWS DMS, Azure Synapse, MS SQL Server, Snowflake, Excel