• Developed data processing pipelines using PySpark such as reading data from external sources, merging the obtained data, performing data enrichment, and loading into Azure SQL Datawarehouse. • Designed and implemented ETL workflows using Spark SQL and PySpark to transform raw data into structured and optimized datasets.• Created and maintained data pipelines using Scala, Python and Spark Streaming to support real-time data ingestion and processing.• Extract Transform and Load data from diverse source systems to Azure Data Storage services using a combination of Azure Data Factory, Spark SQL, and Azure Data Lake Analytics.• Developed the robust ETL pipeline from Azure Blob and Data Lake storage to loading data into Azure SQL Synapse analytics (DW) using Azure HD insights.• Migrated On-premises SQL Server Database to Azure Data Lake, Azure SQL Database, Databricks, and Azure SQL Datawarehouse.• Integrated Custom Visuals based on business requirements using Power BI desktop and worked on building visuals and dashboards using the Power BI reporting tool.

• Implemented advanced data model in Snowflake Datawarehouse to deliver multi-dimensional star and snowflake schemas by normalizing the dimension tables as appropriate.• Maintained and developed complex SQL queries, views, functions, and reports that qualify customer requirements on Snowflake.• Prepared scripts to automate the ingestion and loading process using Airflow as needed through various sources such as Web API, AWS S3, Teradata, and Snowflake.• Scheduled Apache Airflow DAGs to run multiple Hive and spark jobs, which independently run with time and data availability.• Worked on data analysis by using NumPy and Pandas libraries in python to filter unstructured data tostructured format.• Deployed and configured Elasticsearch, Logstash, and Kibana (ELK) for log analytics, full-text search, and application monitoring in integration with AWS Lambda and CloudWatch.• Worked on AWS pipelines for data migration to create and monitor multiple instances such as EC2, S3, AWS lambda, Sage Maker, and EMR.• Designed, developed, and implemented ETL solutions on AWS and in the Big Data environment, as well as migration of existing objects from on-premises, Redshift, and HDFS to S3 Data Lake.• Created automation and deployment templates for relational and non-relational databases including MS SQL, MySQL, and MongoDB can be used in AWS RDS.• Proposed branching strategies for using Version Control Systems like GIT, Stash, GitHub & Subversion. Created and maintained Subversion/GIT repositories, branches, and tags.• Developed and implemented several types of sub-reports, drill-down reports, summary reports, parameterized reports, and ad-hoc reports using Tableau.

• Developed data processing tasks using Pyspark such as reading data from external sources, merging data, performing data enrichment, and loading into target data destinations.• Worked on data analysis and data mining algorithms using Teradata. Loaded the data from Teradata to HDFS using Teradata Hadoop connectors.• Worked on Sqoop for data Ingestion, Hive & Spark for Data Processing & Oozie for designing complex workflows in the Hadoop framework.• Ability to migrate data between RDBMS and HDFS and vice versa using Sqoop.• Implemented Hadoop jobs on an EMR cluster performing several Spark, Hive & MapReduce Jobs for processing data for building recommendation Engines, Transactional fraud analytics, and Behavioral insights.• Hands-on experience in developing web applications and RESTful web services and APIs using Python, Flask, and Django. Experienced in Working on Big Data Integration and Analytics based on Hadoop, PySpark, and No-SQL databases like HBase and MongoDB.• Worked on Hadoop technologies like spark and hive Including using the PySpark library to create spark data frames and converting them to normal pandas’ data frames for analysis.• Worked on moving data from Azure Blob storage data to Azure Data Lake using Azure Data Factory (ADF). • Writing notebooks using PySpark on Azure Databricks to ETL data into Azure Data Lake and designing Azure Data warehouse applications for Medicaid and Medicare data.• Loaded ERP RDBMS data using Sqoop and unstructured machine log data into an AWS S3 Data Lake then into an AWS EC2 Hadoop and Spark Installed Architecture utilizing HDFS, Name Node, Data Node, YARN.• Implemented Data Pipelines using Lambda architecture for batch as well as stream processing using Kafka, Spark, Cassandra, Hadoop, etc. to support online/offline operations.• Developed pipeline using Hive (HQL) to retrieve the data from Hadoop cluster, SQL to retrieve data from Oracle database, and used ETL for data transformation.

• Installed and configured Hadoop MapReduce, HDFS Developed multiple MapReduce jobs in java for data cleaning and preprocessing. Worked on installing, configuring, and using Hadoop ecosystem components. • Worked in administration, installing, upgrading, and managing CDH3, Pig, Hive & HBase Importing and exporting data into HDFS and Hive using Sqoop.• Extraction, Transformation, and Loading (ETL) data into Data Warehouses, as well as data processing tasks such as collecting, aggregating, and moving data from numerous sources using Kafka, Power BI, Microsoft SSIS, MapReduce, PySpark, Spark-SQL, and Pig.• Designed and architected scalable data processing and analytics solutions, including technical feasibility, integration, development for Big Data storage, processing and consumption of Azure data, analytics, big data (Hadoop, Spark), business intelligence (Reporting Services, Power BI), NoSQL, HDInsight, Stream Analytics, Data Factory, Event Hubs, and Notification Hubs.• Hands-on experience in developing web applications and RESTful web services and APIs using Python, Flask, and Django. Experienced in Working on Big Data Integration and Analytics based on Hadoop, PySpark, and No-SQL databases like HBase and MongoDB.• Implemented Hadoop jobs on an EMR cluster performing several Spark, Hive & MapReduce Jobs for processing data for building recommendation Engines, Transactional fraud analytics, and Behavioral insights.