 Worked on end-to-end ETL processes using ADF, ensuring seamless data integration and quality. Hands on experience on creating spark clusters and parameters in the databricks notebooks. I have implemented delta lake concepts in databricks. Experienced in implementing SCD1, SCD2 using delta lake and data validations using pyspark and scheduling databricks notebooks using databricks jobs.Hands on knowledge on using dbutils commands and processing different file formats like csv, parquet, Json etc. Hands on knowledge on different connectors in databricks like creating mount point to connect to data lake storage and creating JDBC connections from databricks to azure synapse Strong knowledge of analytics functions like window, rank and dense_rank etc. and integrating databricks with azure key vault. Developed pyspark code for data cleansing like trimming of columns and duplicate checks, key duplicates etc. and SCD Type1 using merge functionality. Hands on experience in implementing different types of joins in pyspark and SQL. Having experience in implementing Azure Data factory(v2) pipeline components such as linked services, Datasets and Activities. Implemented multi tables full load from on-premises to cloud and incremental load from on-premises to cloud and Implemented Audit in Azure Data Factory using stored procedure. Implemented the scheduling in Azure Data Factory using Triggers and Dynamic Data loading using a Config. Table using components Lookup, Foreach and Copy Data activities for the tables which are configured in the Config. Table and experience on Key Vaults. Scheduling ADF pipelines using scheduled triggers and event-based triggers. Working experience Pipelines in ADF using Linked Services/Datasets/Pipeline to Extract and load data from different sources like Azure SQL, ADLS, Blob storage, Azure SQL Data warehouse. Involved in Bug fixes and code debugging and job monitoring in Production Environment.

 Established and developed serverless applications on AWS using the Serverless Framework and Python's boto3 module. Build serverless applications utilizing AWS Lambda, API Gateway, and DynamoDB, resulting in a reduction in infrastructure costs and an improvement in scalability. Designed and developed end-to-end data pipelines using AWS Glue, Apache Airflow and Apache Spark to extract, transform, and load data from various sources into data warehouses. Integrated data quality checks and data governance mechanisms to ensure data accuracy and consistency throughout the organization. Created and designed ETL processes in AWS Glue to import various kinds of from outside sources into AWS Redshift. Monitored and maintained the health and performance of Amazon Redshift clusters through monitoring tools and dashboards. Managed backups and disaster recovery procedures to ensure data integrity and business continuity. Implemented data encryption and security measures to protect sensitive data in compliance with industry standards. Designed and optimized data models, including defining schemas and setting up indexes, to enhance data storage and retrieval efficiency in Snowflake. Developed and maintained data pipelines using Apache Airflow and custom scripts to extract, transform, and load (ETL) data from diverse sources into the RedShift data warehouse. Set up proactive monitoring and alerting systems to detect and address data loads, query performance, and system health issues. Utilized Apache Spark for distributed computing tasks, improving processing speeds.

 Responsible for creating a data lake on the Azure Cloud Platform to improve business teams' use of Azure Synapse SQL for data analysis. Utilized Azure SQL as an external hive meta store for Databricks clusters so that metadata is persisted across multiple clusters. Employed Azure Data Lake Storage as a data lake and made sure that spark and hive tasks immediately sent all the processed data to ADLS. Strong Experience working with Azure Databricks runtimes and utilizing data bricks API for automating the process of launching and terminating runtimes. Experience in integrating Snowflake data with Azure Blob Storage and SQL Data Warehouse using Snow Pipe. Employed resources like SQL Server Integration Services, Azure Data Factory, and other ETL tools to identify the route for transferring data from SAS reports to Azure Data Factory. Developed PowerShell scripts for automation and configuration management. Implemented data processing workflows using PySpark, leveraging the power of Python for Spark. Developed and optimize Spark pipelines for efficient large-scale data processing.  Designed and deployed infrastructure using Azure Resource Manager (ARM) templates. Implemented and managed SQL Data Warehousing (SQL DW) solutions for analytical processing.  Maintained Azure Data Factory to consume data from several source systems and transferred data from upstream to downstream systems using Azure Data Factory as an orchestration tool.  Developed a Pipelines in Azure Data Factory (ADF) using Linked Services/Datasets/Pipeline to extract, transform, and load data from a variety of sources, including Azure SQL, Blob storage, an Azure SQL data warehouse, write-back tools, and backwards.  Implemented and optimized data storage solutions using Azure Cosmos DB. Utilized Azure Analysis Services for multidimensional data analysis and reporting.

 Developed ETL jobs using Spark -Scala to migrate data from Oracle to new MySQL tables. Rigorously exerted Spark -Scala (RDD’s, Data frames, Spark SQL) and Spark - Cassandra -Connector APIs for various tasks (Data migration, Business report generation etc.) Developed Spark Streaming application for real time sales analytics. Prepared an ETL framework with the help of Sqoop, pig and hive to be able to frequently bring in data from the source and make it available for consumption. Processed HDFS data and created external tables using Hive and developed scripts to ingest and repair tables that can be reused across the project. Engineered complex data pipelines using tools such as Databricks, processing terabytes of data to drive decision-making. Analyzed the source data and handled efficiently by modifying the data types. Worked on excel sheet, flat files, CSV files to generated PowerBI ad-hoc reports. Analyzed the SQL scripts and designed the solution to implement using PySpark. Extracted the data from other data sources into HDFS using Sqoop. Handled importing of data from various data sources, performed transformations using Hive, MapReduce, loaded data into HDFS. Extracted the data from MySQL into HDFS using Sqoop. Implemented automation for deployments by using YAML scripts for massive builds and releases. Apache Hive, Apache Pig, HBase, Apache Spark, Zookeeper, Flume, Kafka and Sqoop. Implemented Data classification algorithms using MapReduce design patterns. Extensively worked on creating combiners, Partitioning, distributed cache to improve the performance of MapReduce jobs.  Worked on GIT to maintain source code in Git and GitHub repositories.

* Worked with Stakeholders regarding business requirements, functional specifications and enhancements, based on the business needs created technical design and functional specification documents.*Data profiling the source files and developing data model and mappings for smaller requirements.*Implemented CDC, SCD2, SCD1 Delta load, Snapshot and transactional fact tables, headers and footers to Flat File, File list.* Active participation in weekly calls with data modeling and analyst teams to understand and work on any new requirements.*Analyzing data from source systems to design the solution for the business requirement.*Developed Complex Mappings in Informatica using Power Center transformations (Source Qualifier, Joiner, Lookups, Filter, Router, Aggregator, Expression, XML Update and Sequence generator transformations), Mapping Parameters/Variables, Parameter files, SQL overrides, Transformation Language.*Developed Unix scripts for SFTP file transfers and Target table truncate operations.*Implemented partitioning at database level for better performance.*Implemented Push Down Optimization (PDO) for better performance when source data is huge.*Provided support to the QA team for various testing phases of ETL development.*Scheduled the Workflows to run on a daily and weekly basis using Control-M Scheduling tool.