Software Engineer Iii - Data Engineer
Bengaluru, Karnataka, India
- Designed and led Apache Spark-based generic data loader framework, helped in reducing overall effort spent on building a batch job by 90%, enabled data loads from multiple sources to targets by applying custom.
- Led a team of 4 in NodeJS-based backend development for an ML-driven application, utilizing the Backend For Frontends microservice design pattern.Directed team in designing and implementing the application's database.
- Managed a data engineering team of 7 and migrated a sales forecasting project from Azure to GCP, collaborated with the operations team, and scheduled monthly level runs using Apache Airflow
- Developed a GitOps pipeline that uses DDL and DML scripts versioned in the GitHub repository to deploy tables in Google BigQuery target environments ( dev, stage, prod ). The pipeline template was leveraged across.
- Redesigned and enhanced Walmart's patented Travel Bot project by migrating its data sources from Hadoop and Hive to Azure SQL; further innovated by developing a generic parser to extract diverse airline contracts for.