Data Engineer
Current- Increased data reliability and stakeholder satisfaction by 25% by improving Airflow task scheduling reliability
- Developed new functionality to enhance ETL batch processing Data Ingestion, improving operational efficiency & reducing manual workload by 40%.
- Reduced costs for AWS EMR service by 60% by optimising infrastructure provisioning & auto-scaling.
- Designed and developed real-time Spark streaming pipelines, handling peak rates of 10k messages/sec
- Improved operational efficiency, monitoring & alerting by 50% through Cloudwatch & PD Alarms
- Recipient of quarterly award for demonstrating pro-activeness in improving processes and exceeding expectations.