Big Data Engineer
Current- Developed and maintained data processes using Airflow, managing over 100 ETL jobs daily for smooth pipeline execution across multiple markets.
- Utilized ServiceNow ticketing system to address and resolve an average of 20 production issues weekly, achieving a 99.9% uptime and minimizing disruptions.
- Automated data loading processes through Linux shell scripts, increasing efficiency by 40% and reducing manual intervention by 50 hours per week.
- Spearheaded migration from Airflow1 to Airflow2 with zero downtime, ensuring uninterrupted data processing.
- Leveraged Python, SQL, HDFS, Linux, MariaDB, and PySpark to optimize data processes, resulting in a 30% improvement in processing speed.
- Engaged in daily calls with global team members, providing instructions to the monitoring L1-support team and addressing client data-related issues promptly.