Developed report scheduling platform for 200+ analystsMentored 10+ budding data engineersAutomated infrastructure workflows for EMR, Kafka, and Airflow.Lead and developed a single stop portal for all data needs such as new syncing requests, report scheduling, Kafka topic creation, aggregate table creation, etc. For reference, new table sync could now be achieved in under an hour.Enhanced our ELT pipelines by adding features such as auto schema evolution, view-based table persistence, support for Confluent Avro as a data source, etc.Worked on 100+ data pipeline setupsMigrated services (Presto, Kafka Connect, Presto Gateway, Metabase, Kafka services) from EC2 to Kubernetes. This lead to better load and cost efficiency with easy scale up and scale down of services

Worked on verification of an SVM-based algorithm that minimized network traffic between power stations by optimally predicting metrics and auto-correcting when changes exceeded specified error threshold.