• Implemented Azure Data Factory (ADF) extensively for ingesting data from different source systems like structured and unstructured data to meet business functional requirements.• Implemented the data warehousing solution in Azure Synapse Analytics.• Developed Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transformand load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards• Implemented performance tuning on the spark applications in DataBricks notebooks which improved the overall performance by 5 times than the original jobs.• Built complex ETL/ELT pipelines for data processing in the azure cloud using Azure Data Factory V2, and Azure Synapse Dedicated SQL Pools.• Have good experience working with Azure BLOB and Data Lake storage and loading data into Azure SQL Synapse analytics (DW).• Using Azure Synapse pipeline capabilities, CDC (Change Data Capture logic) was implemented for incremental data loading.• Developed the ELT processing framework using Azure Synapse Analytics, completing an end-to-end migration project from on-premise to Azure cloud.• CDC (which is now in public preview from Azure) is being tested in order to identify and record any changes made to our relational databases.• Pipelines were developed to transfer hashed and unhashed data from Azure Blob to Data Lake.• Using Azure Data Factory pipelines, we back traced the ETL/ELT mappings for dataflows and enhanced data processing performance overall.• To load the data into the fact and dimension tables in Azure SQL Datawarehouse, stored procedures were created.• Developed sophisticated queries using PySpark/Spark SQL on Azure Synapse Spark pools according to the business requirement.• Scheduled and automated complex ETL workflows using Airflow’s Cron expressions to trigger data pipelines at specified intervals, ensuring timely data processing.

• Use Azure PaaS services to analyze, create, and construct contemporary data solutions that facilitate data visualization. Recognize the application's present production state and assess how a new implementation may affect ongoing business procedures. • Utilizing Azure Data Factory, T-SQL, Spark SQL, and U-SQL Azure Data Lake Analytics, extract, transform, and load data from Sources Systems to Azure Data Storage services. • JDBC was used for database connectivity, while Liquidate was used for database migration.• Data processing in Azure Databricks after being ingested into one or more Azure Services (Azure Data Lake, Azure Storage, Azure SQL, Azure DW).• Designed and implemented data pipelines to migrate large datasets from Oracle databases to Azure Data Lake and Azure SQL Database using Azure Data Factory (ADF), ensuring smooth transition and data integrity.• Using Linked Services/Datasets/Pipeline, I developed pipelines in ADF to extract, transform, and load data from many sources, including Azure SQL, Blob storage, Azure SQL Data Warehouse, write-back tool, and backwards.• Used Azure Service Bus to enable real-time data processing and improve system reliability.• Worked in developing scripts using Python, PySpark, Shell Scripting to do Extract, Load, and Transform data working knowledge of Azure Databricks.• Completed online data transfer from AWS S3 to Azure Blob by using Azure Data Factory.• Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.• In control of tracking, troubleshooting, and cluster size estimation for the Spark data bricks cluster.• Working with Azure Data Lake and Snowflake cloud data warehouse for integrating data from various source systems, including importing nested JSON-formatted data into Snowflake tables.

• Built and maintained big data pipelines using Apache Spark and for processing large-scale datasets from multiple sources.• Designed and developed ETL processes using Pig scripts and HiveQL to clean, transform, and load data into the data warehouse.• Leveraged Hive for querying and analyzing massive datasets, improving business reporting and decision-making processes.• Implemented Spark Streaming for real-time data ingestion and processing from streaming sources.• Collaborated with data scientists and analysts to provide them with clean, structured data for analysis and predictive modeling.• Worked with large-scale Hadoop clusters to process and analyze data using MapReduce, Pig, and Hive.• Developed and deployed Spark jobs for batch processing of structured and unstructured data, improving data processing efficiency.• Created Pig scripts to automate data transformation tasks, reducing manual efforts in data cleaning and preparation.• Managed data storage and retrieval using HDFS, ensuring high availability and fault-tolerant data storage.• Supported data warehousing solutions by developing data pipelines that integrated data from multiple sources, including relational databases and log files.• Wrote HiveQL queries to generate insights from large datasets for reporting and business intelligence purposes.• Assisted in building and maintaining Hadoop clusters for big data processing.• Supported the development of ETL workflows using Hive and Pig for extracting and transforming data from various sources.• Configured PyTest for testing Python functions used in data pipelines, improving the reliability and consistency of the data engineering codebase.• Wrote and optimized MapReduce jobs to process and analyze large datasets in Hadoop.• Collaborated with senior data engineers and analysts to ensure data pipelines met business requirements and performance standards.

• Designed and optimized SQL queries to extract, manipulate, and analyze large datasets from SQL Server and PostgreSQL, supporting key business initiatives.• Developed complex data models and automated ETL pipelines to integrate data from multiple sources, ensuring accurate and consistent reporting.• Created dynamic dashboards and reports using Power BI by querying datasets in real-time, enabling stakeholders to monitor performance metrics and KPIs.• Collaborated with business stakeholders to understand requirements and translate them into actionable SQL-based reports for marketing, sales, and finance teams.• Automated routine reports using SQL and stored procedures, reducing manual work and ensuring timely delivery of insights• Developed SQL queries to aggregate and analyze transaction data, identifying trends and patterns in customer behavior to enhance marketing strategies.• Led data migration projects from legacy systems to SQL Server, optimizing data storage and retrieval processes.• Performed data validation and cleansing using SQL functions and scripts, ensuring accurate, high-quality data for reporting purposes.• Built and maintained ad-hoc reports for executive leadership, utilizing complex SQL queries, including window functions and common table expressions (CTEs), to provide deep insights into sales, inventory, and customer data.• Collaborated with senior engineers to write PyTest test cases, ensuring data integrity in simple data workflows and helping to detect bugs early in the development process.• Monitored and optimized database performance, ensuring fast and efficient data processing.• Assisted in creating SQL queries for routine reports and analytics, including customer segmentation and product performance analysis.• Supported the ETL team by writing SQL scripts for data extraction, transformation, and loading into data warehouses.