• Involved in Requirement gathering, business Analysis, Design and Development, testing and implementation of business rules.• Understand business use cases, integration business, write business & technical requirements documents, logic diagrams, process flow charts, and other application related documents.• Used Pandas in Python for Data Cleaning and validating the source data.• Created pipelines, data flows and complex data transformations and manipulations using Glue and PySpark.• Developed ETL applications using Python, Spark (PySpark) and Shell scripting based on the business requirements.• Involved with Data Profiling for multiple sources and answered complex business questions by providing data to business users.• Wrote complex SQL queries for validating the data against different kinds of Database systems to reconcile data across systems such as Snowflake and Oracle.• Developed CI/CD system with Jenkins on Docker for the runtime environment for the CI/CD system to build, test and deploy.• Worked on PySpark jobs and troubleshooting of PySpark job performance. Worked on Batch Data pipelines using PySpark and Scala• Good experience working with Databricks Notebook platform and Snowflake Cloud Data Warehouse systems.• Worked on building centralized Data Lake on AWS Cloud by utilizing services like SQS, SNS, S3, EMR, RedShift and Athena.• Leveraged snowflake’s Snow Sight and Snow SQL to ingest and build data models for downstream analytics.• Worked on Data Ingestion from external systems into S3 data lake using python and boto3 module.• Worked on Docker files to containerize the spark application.

• Involved in writing PySpark applications to perform various data cleansing, validation, transformation, and summarization activities according to requirement.• Responsible for developing data pipelines using Azure data factory involving investigating raw JSON files, transactional and user profile information from on prem data warehouses and processing them using PySpark and Scala.• Written Python, PySpark, Scala transformations using Jupyter and Databricks on data loaded in Azure Data Lake Storage• Using Synapse Dedicated SQL pool Connecter for writing the processed data from spark to dedicated SQL pool.• Written python and spark transformations on raw data to clean.• Written python validation code to check consistency of ingesting data.• Worked Extensively on improving performance and troubleshooting data pipelines in spark and Delta Tables.• Implemented dockized spark notebook which support docker containerization.• Involved in deployment of Docker Container into Kubernetes clusters. • Implemented the migration of data from multiple on-premises servers into cloud using Data Factory service in Azure and Data Migration Assistant application.• Implemented Azure Synapse pipelines to gather data from SQL On-prem and cloud Databases.• Worked with SSMS & SSIS packages to extract data from on-prem databases.• Worked PII data encryption SSIS packages while doing from customer facing databases.• Worked on data analytics with Power BI to check data quality and correctness.• Worked on Azure Event grid to collect data from sensors.• Implemented Azure functions and logic apps to run serverless apps.• Worked with Hive and SQL Server to store and capture data using data pipelines.• Written spark transformations in synapse notebooks to clean the raw data.• Worked on different compressed file formats like Parquet, AVRO, ORC using spark.• Used Snowflake as a Data warehousing system to store processed data from Databricks and ADF.

• Implemented End to End Unity Catalog on top of Azure Databricks.• Working on implementation of Databricks workflows for batch data pipelines.• Handled compute workload optimization by utilizing the spark and databricks optimizations and improved efficiency by 30% and lowered cost.• Lead and implemented End 2 End Accident and Health project from gathering business requirements to implementation in PySpark using Spark and Spark SQL.• Utilized Azure Data Factory generic pipelines across multiple projects to ingest data from various platforms like SFTP network shares, On prem SQL servers, S3, DB2.• Implemented medallion three zone data transformation and cleansing and storage in gold layer.• Worked with synapse external tables to load databricks tables for analytical purposes.• Built series of PySpark Applications and Hive scripts to produce various analytical datasets needed for analytical and BI teams.• Involved in deployment of Docker Container into Kubernetes clusters. • Used terraform scripts for deployment of workloads across multiple environments.• Built ETL pipelines using Azure Data Factory and Databricks to ingest data from On-prem Servers to Azure Data Lake.• Worked with SSIS packages on Visual Studio to extract data from databases while maintaining change data capture updates.

• Developing and maintaining a Data Lake containing regulatory data for federal reporting with big data technologies such as Hadoop Distributed File System (HDFS), Apache Impala, Apache Hive and Cloudera distribution.• Developing different ETL jobs to extract data from different data sources like Oracle, Microsoft SQL Server, transform the extracted data using Hive Query Language (HQL) and load it into Hadoop Distributed file system (HDFS).• Primarily involved in Data Migration process using AWS by integrating with Github repository and Jenkins.• Designed, developed and did maintenance of data integration programs in a Hadoop and RDBMS environment with both traditional and non-traditional source systems as well as RDBMS and NoSQL data stores for data access and analysis.• Implementing new functionality in the Data Lake using big data technologies such as Hadoop Distributed File System (HDFS), Apache Impala and Apache Hive based on the requirements provided by the client.• Developing Python scripts using Hadoop Distributed File System API’s to generate Curl commands to migrate data and to prepare different environments within the project.

• Involved in Requirements Analysis and design and Object-oriented domain model.• Involved in development of Application with Java and J2EE technologies.• Develop and maintain elaborate services-based architecture utilizing open-source technologies like Hibernate, ORM and Spring Framework.• Developed server-side services using Java multithreading, Struts MVC, Java, EJB, Spring, Webservices (SOAP, WSDL, AXIS).• Responsible for developing DAO layer using Spring MVC and configuration XMLs for Hibernate and to also manage CRUD operations (insert, update, and delete).• Deployed Web, presentation, and business components on Apache Tomcat Application Server.