Skilled in developing Spark applications using PySpark and Spark-SQL for data extraction, transformation, and aggregation, enabling insights into customer usage patterns. Proficient in utilizing Agile/SCRUM methodologies to drive software development cycles from requirement gathering to deployment. Expertise in Java Spark, Splunk, and AWS, with a proven track record of building scalable and efficient data processing pipelines in cloud environments. Demonstrated success in migrating on-premises ETL processes to the cloud and deploying and maintaining data pipelines on AWS using Glue, Athena, Kinesis, Lambda, and Step Functions. Experienced in managing databases using AWS RDS and Amazon Aurora, and proficient in migrating databases to Amazon Aurora with minimal downtime. Strong proficiency in AWS services such as S3, Lambda, and Glue for scalable data storage and processing. Skilled in integrating data from multiple sources into data marts and utilizing shell scripts for automation. Experienced in Microsoft Azure DataBricks and Azure Data Factory, Snowflake data warehousing, and symmetric/asymmetric encryption. Proven ability to develop API endpoints, handle various file formats, and optimize data processing workflows for efficiency. Adept at using VSTS for source code management and committed to delivering high-quality solutions to meet business needs.

• Developed and designed data integration and migration solutions in Azure.• Kept our data separated and secure across national boundaries through multiple data centers and regions.• Create Spark Vectorized panda user defined functions for data manipulation and wrangling.• Migrating on-perm ETLs from MS SQL server to Azure using Azure Data Factory and Azure Data bricks.• Setting up Azure infrastructure like storage, integration runtime, service principal id, app registrations to enable scalable and optimized utilization of business user analytical requirements in Azure.• Create and maintain optimal data pipeline architecture in Microsoft Azure using Data Factory and Azure Databricks.• Creating Data factory pipelines that can bulk copy multiple tables at once from relational database to Azure data lake gen2• Migrate data into RV Data Pipeline using Databricks, Spark SQL and Scala.

• Environment Developed and maintained a regulatory data lake for federal reporting using big data technologies such as Hadoop Distributed File System (HDFS), Apache Impala, Apache Hive, and Cloudera Distribution.• Involved in importing data from various sources into HDFS using Sqoop, applying transformations using Hive, Spark, and loading data into Hive tables.• Primarily involved in the data migration process using AWS with GitHub repositories and integration with Jenkins.• Design, develop and maintain data integration programs in Hadoop and RDBMS environments. Using traditional and non-traditional source systems and his RDBMS and NoSQL data stores for data access and analysis.• Develop Python scripts using the Hadoop Distributed File System API to generate curl commands to migrate data and prepare various environments within a project.

• Developing different ETL jobs to extract data from different data sources like Oracle, Microsoft SQL Server, transform the extracted data using Hive Query Language (HQL) and load it into Hadoop Distributed file system (HDFS).• Experience in using Sqoop for Importing and exporting data into HDFS and Hive.• Primarily involved in the Data Migration process using AWS by integrating with GitHub repository and Jenkins.• Experienced in handling data from different data sets, and using Pig for data joins and data preprocessing operations.• Primarily responsible for designing, implementing, Testing, and maintaining database solution for AWS.

• Implemented Partitioning, Dynamic Partitions and Buckets in HIVE for efficient data access. • Create/Modify shell scripts for scheduling various data cleansing scripts and ETL load process.• Responsible for importing data to HDFS using Sqoop from different RDBMS servers and exporting data using Sqoop to the RDBMS servers after aggregations for other ETL operations.• Involved in Functional Testing, Integration testing, Regression Testing, Smoke testing and performance Testing. Tested Hadoop Map Reduce developed in python, pig, Hive.• Experience in designing and developing applications in PySpark using python to compare the performance of Spark with Hive.