Data Engineer
◦ Worked as Data Engineer under Accenture company and developed ETL workflows using Talend and Databricksfrom various sources into RDS(MySQL) and delta lake.◦ Worked on crucial modules in healthcare domain like Networks, Formulary, Plans, Eligibility, Claims and Phase module.◦ Implemented 12 new clients,15 Plan year updates,20 PBM Switch tasks and resolved more than 50 bug tickets in both Talend and Databricks together.◦ Automated data cleansing process using PySpark and JSON in Azure databricks, resulting in 80% reduction in manual effort and automated Production workflows using Azure Data Factory.◦ Experience in building Real-time Data Pipelines with Kafka Connect and Spark Streaming.◦ Optimized performance for production member data management workflow in Databricks by 60% compared to Talend by reducing runtime from 9 hrs to 3 hrs and zero manual maintenance on Infrastructure.◦ Reduced Data clean ups with code optimization and extended data quality checks to ensure data accuracy, consistency, and completeness.◦ Worked on migrating more than 80 pipelines from Talend to Databricks and collaborated with cross-functional teams while designing workflows on databricks during migration from Talend.◦ Create several types of data visualizations using Python and Tableau.◦ Developed DMS task that will load data from RDS to Delta Lake using AWS and Delta Live Tables (DLT).◦ Been a part of peer programming with other team members in critical scenarios and provided knowledge transfer on design flow in databricks for support teams.◦ Designed and implemented test cases covering various scenarios including boundary cases, edge cases, and error conditions to validate the functionality of SQL code.
