• Developed and maintained scalable data pipelines using Snowflake, FiveTran, and custom ETL processes, ensuring efficient data extraction, transformation, and loading.• Designed and implemented data models to support various analytical and operational requirements, optimizing performance and storage utilization.• Collaborated with cross-functional teams to gather requirements, define data integration strategies, and deliver solutions aligned with business objectives.• Implemented data quality checks and monitoring mechanisms to ensure the accuracy, completeness, and timeliness of data across multiple sources.• Utilized Alation for metadata management, data cataloging, and lineage tracking, improving data governance and compliance.• Automated deployment and orchestration of data pipelines using DevOps tools such as Jenkins, GitLab CI/CD, and Docker, reducing manual effort and improving reliability.

• Utilized Python, SQL, and PySpark within Azure Databricks to process streaming data in real-time, applied complex transformations and enrichments to ensure data quality and relevance.• Creating sandbox Data Factory for developing data pipelines with Datasets and Linked Services.• Developing Fact and Dimension tables & views for various markets. • Extracting and transforming data using Databricks and Data Factory.• Creating and mataining complete lifecycle of the data till presented layer and creating views.• Developed efficient data ingestion pipelines using Azure Data Factory to seamlessly ingest data from Azure Blob Storage, Azure SQL Database, and external APIs.

• Transform the raw data into meaningful information to support business decisions.• Develop, Deploy and Troubleshoot the ETL Work Flows using Hive, Pig, Sqoop and Spark.• Develop Map Reduce jobs for Data Cleanup in Python, Java, and C#.• Dimensions and Facts Design and Data Modeling for reporting purposes.• Design and Implement DataMart and improve performance.• Identify the Reports for decision-making and create the Dash Boards by using the power.• Tested and solved the critical issues faced while data ingestion into the data unit.

• Involved in designing, developing Pyspark Notebook in Azure Databricks and visualizing the data in PowerBI.• Prepared the complete data flow and data mapping for all events generated by the Legacy system.• Involved in Data Analysis for creating the Target Json Model, which is stored in Azure Cosmos DB and involved in one-time data load. • Created the framework for bringing the data from Legacy System into Data Lake and mounted in Azure Databricks as Delta table in Raw Layer.• Created the framework in Transforming Raw Layer data into Standardization Layer.• Created Optimized View in Delta table required for PowerBI report.• Involved in performance tuning in Delta tables and PowerBI report. • Created PowerBI report from the Databricks Delta tables.• Develop project artifacts (e.g. standards & guidelines, program specifications, test strategy, test plans) as per the defined schedule and estimates.

• Analyzing the data through scripting Hive Queries as per client requirements.• Designed and created Hive external and internal table with partitioning and buckets.• Writing the script files for processing data and loading to HDFS.• Involved in converting Hive/SQL queries into Spark transformation using Spark RDD and Pyspark.• Created Spark applications to process the transactional Data using Spark high level APIs such as Dataframes, Spark SQL and Datasets and low level api RDDs in Azure Databricks Environment .• Great understanding of end-to-end scalable architecture to solve business problems using various Azure Components like HDInsight, Data Factory, Data Lake, Storage and Machine Learning Studio.