• Design and implement end-to-end data solutions (storage, integration, processing, visualization) in Azure and Databricks.• Responsible in creating the technical design document (TDD) related to the overall architectural data workflow.• Responsible in planning and defining the load strategies based on the resources and business requirement.• Architect and implement ETL and data movement solutions using Azure Data Factory.• Developed the plans to migrate the data from on premise SQL Server to Azure Data Lake Store (ADLS) using Azure Data Factory.• Developed the ADF pipelines using the copy activity, Custom Azure Data Factory Pipeline Activities for On-cloud ETL processing.• Responsible in creating the Linked Services, Datasets, Pipelines to Extract, Transform and load data from different sources like Azure SQL, Blob storage and Azure SQL Data warehouse.• Conducting the analysis and based upon the requirement, pipeline run interval, and business data consumption, the Event-based triggers are created in Azure Data Factory.• Established the connection to variety of resources like On-Premise SQL Server by setting up Self-hosted Integration runtime in Azure Data Factory.• Created data integration and technical solutions for Azure Data Lake Analytics, Azure Data Lake Storage, Azure Data Factory, Azure SQL databases and Azure SQL Data Warehouse for providing synapse analytics and reports for improving marketing strategies.• Transformed the raw and semi structured data using the PySpark in databricks and loaded the data back to ADLS Gen2.• Used various Spark Transformations and Actions for cleansing the input data.• Used the Spark SQL to process and parse different data formats like Text, CSV files received from the third-party vendors.• Developed the suitable logic to handle the bad, null, and mal formed data by defining the techniques like failfast, badrecordpath, bad data flag and drop bad data.

• Implemented Azure Data Factory (ADF) extensively for ingesting data from different source systems like relational and unstructured data to meet business functional requirements.• Worked on developing and setting up snowpipe to ingest files in near real time to snowflake tables from DataLake.• Design and developed Batch processing and real-time processing solutions using ADF and Databricks clusters• Created numerous pipelines in Azure using Azure Data Factory v2 to get the data from disparate source systems by using different Azure Activities like Transform, Copy, for each, Databricks etc.• Maintain and provide support for optimal pipelines and complex data transformations and manipulations using ADF and PySpark with Databricks.• Automated jobs using different triggers like Events, Schedules and Tumbling in ADF.• Created, provisioned different Databricks clusters, notebooks, jobs and autoscaling.• Implemented Lakehouse using Azure databricks and databricks delta tables.• Developed custom python scripts that helps with data quality issues, emails, data cleansing, reports...etc. • Implemented self-hosted integration runtime to access private network data using Data Factory. • Used Azure Logic Apps to develop workflows that can send alerts/notifications on different jobs in Azure.• Implemented both ETL and ELT architectures in Azure using Data Factory, Databricks, SQL DB, and ADLS.• Experienced in developing audit, balance, and control frameworks using SQL DB audit tables to control the ingestion, transformation, and load process in Azure.• Created Linked services to connect the external resources to ADF.• Working with complex SQL views and Stored Procedures in large databases from various servers.• Worked with team members to resolve any technical issues, Troubleshooting, Project Risk & Issue identification, and management.• Created build and release pipelines for databricks notebooks and data factory pipelines using Azure DevOps.

• Involved in creating mapping specification documents based on design documents.• Develop Databricks Python notebooks to Join, filter, pre-aggregate, and process the files stored in Azure data lake storage.• Implemented both ETL and ELT architectures in Azure using Data Factory, Databricks, SQL DB and SQL Data warehouse.• Used Databricks to ingest data from ADLS and EventHub in near real time using Spark Structured Streaming.• Worked on ingesting data and implementing custom transformations in PostgreSQL using Databricks, Spark and psycopg2 python library.• Worked on setting up custom python logging functions to application insights from Azure Databricks.• Created Integration between control-m and Azure Data Factory to Orchestrate ETL/ELT pipelines.• Created DataFrames, Spark SQL and User Defined Functions to massage the data within the DataLake.• Created pipelines, data flows and complex data transformations and manipulations using ADF and PySpark with Databricks.• Implemented alerting mechanism from Azure Monitor using Logic App and Outlook Connector.• Worked on Teradata BTEQ, Fast Export, and TPT scripts for extracting and loading the data to target tables.• Developed ETL Pipelines that brings and transforms the huge volumes of data from different source systems using ADF and Databricks. • Worked on developing mappings using various transformations like Lookup, Filter, Expression, Aggregator, Joiner, etc., for implementing the ETLs.• Worked on performance tuning Teradata SQLs for various applications to improve the SLA’s.• Coordinating with the team on the development and support activities.• Communicating the critical information related to the project activities across all the application teams.• Created Deployments using Jenkins to deploy several cloud data services/pipelines.

• Gathered requirements from business analysts for designing and development of the system.• Involved in designing Source to Target mapping documents and loading strategies as a part of requirements gathering.• Extensively used ETL to load data from RDBMS, Flat files, XML files, Oracle and legacy data as sources, Flat files as targets.• Developed ETL pipelines using python with pandas and csv modules.• Orchestrated shell and python scripts using crontab in Linux server• Worked with Data Quality group to identify and research data quality issues.• Created a cleanup process for removing all the Intermediate temp files that were used prior to the loading process using shell scripting.• Developed python scripts that helps with migration from one system to other, generate SQL and modify syntax that supports with Target system. • Extensive experience in Performance Tuning -Identified and fixed bottlenecks and tuned the complex Informatica mappings for better Performance.• Involved in the debugging of the mappings by creating break points to gain trouble shooting information about data and error conditions.• Developed various scripts for Teradata utilities.• Developed Unit Test Cases to ensure successful execution of the data loading processes.• Assisted QA team to fix and find solutions for the production issues.