Developed a versatile data ingestion tool in Pyspark, enabling seamless integration from diverse sources like API, RDBMS, Files, and Kafka.• Optimized SQL queries by implementing partitioned table reading, enhancing performance through parallel processing.• Created Pyspark and SQL scripts on AWS Databricks to generate state-specific health reports, critical for risk analysis and customer service optimization.• Designed the ETL processes using the Informatica tool to load data from Oracle, flat files into the target Oracle Database.• Spearheaded AWS Data pipeline configuration for efficient data loads from S3 to RedShift, contributing to scalable data solutions on Amazon EMR.• Responsible for continuous integration (CI) and continuous delivery (CD) process implementation using Jenkins along with Shell scripts to automate routine jobs.• Managed data importing and transformations from various sources, utilizing Spark and storing results in Hive and S3 buckets.• Scheduled Airflow jobs, orchestrating multiple Spark jobs with shell scripts for efficient resource utilization.• Designed, developed, and deployed software solutions for clients using Python and Django, ensuring clean, maintainable, and scalable code following best practices.• Worked on Jenkins file with multiple stages like checkout a branch, building the application, testing, pushing the image into GCR, Deploying to QA3, Deploying to QA9, Acceptance testing and finally Deploying to Prod.• Developed SQL, PL/SQL, and stored procedures, accompanied by Shell Scripting for robust database applications.• Experienced in Docker container, writing Docker files, Docker compose, Docker network, and private Docker repo.• Developed Kafka consumer API in Python for consuming data from Kafka topics.

•Experienced in translating business users needs into ETL Technical Design Documents. Collaborated with Teams from concept through final delivery of product.• Worked on AWS Redshift, S3 and Athena services to query large amounts of data stored in S3 to create a virtual Data lake without going through the ETL Process.• Developed ETL procedures to migrate data from different sources to the data lake.• Data Extraction, aggregations, and consolidation of Adobe data within AWS Glue using PySpark.• Developed file cleaners using Python libraries and utilized Python libraries Boto3, Pandas• Proficient in conducting root cause analysis of data break issues utilizing SQL, PLSQL, and Unix Code debugging techniques, ensuring prompt resolution and minimizing downtime.• Designed and implemented a configurable data delivery pipeline for scheduled updates to customer-facing data stores built with Python.• Managed the Transaction module while developing and testing Oracle, PL/SQL stored procedures, functions, and triggers.• Developed Shell scripts to automatically backup repositories whenever new revisions are made and send emails to respective owners.• Created a test automation framework to minimize the coding effort and ease the process of maintenance.• Successfully led and executed a complex database migration project, transferring data and schemas from SQL server databases to Oracle, ensuring data integrity and minimal downtime during the transition.• API builds, deployments, and configuration management. Troubleshoot issues related to build and deployment failures - worked on Jenkins and TeamCity.• Played a key role in issue resolution by analysing log files, debugging code, and collaborating with engineers, ensuring customer issues were resolved within SLA timeframes.• Collaborated with product architects, software developers, and QA engineers to plan test automation priorities and define overall technical approaches and plans.