Data Engineer
Current• Increased data reliability and stakeholder satisfaction by 25% by improving Airflow task scheduling reliability • Developed new functionality to enhance ETL batch processing Data Ingestion, improving operational efficiency & reducing manual workload by 40%.• Reduced costs for AWS EMR service by 60% by optimising infrastructure provisioning & auto-scaling. • Designed and developed real-time Spark streaming pipelines, handling peak rates of 10k messages/sec • Improved operational efficiency, monitoring & alerting by 50% through Cloudwatch & PD Alarms• Recipient of quarterly award for demonstrating pro-activeness in improving processes and exceeding expectations.• Became sole owner of ad revenue Data Ingestion & held deep-dive knowledge sharing session on how ad revenue data is ingested from various sources (APIs, s3, GCS, Emails) in various formats