Big Data Engineer
Current• Developed and maintained data processes using Airflow, managing over 100 ETL jobs daily for smooth pipeline execution across multiple markets.• Utilized ServiceNow ticketing system to address and resolve an average of 20 production issues weekly, achieving a 99.9% uptime and minimizing disruptions.• Automated data loading processes through Linux shell scripts, increasing efficiency by 40% and reducing manual intervention by 50 hours per week.• Spearheaded migration from Airflow1 to Airflow2 with zero downtime, ensuring uninterrupted data processing.• Leveraged Python, SQL, HDFS, Linux, MariaDB, and PySpark to optimize data processes, resulting in a 30% improvement in processing speed.• Engaged in daily calls with global team members, providing instructions to the monitoring L1-support team and addressing client data-related issues promptly.• Implemented over 50 modifications to Airflow DAG codes, enhancing workflow efficiency by 20% and reducing processing time by 15%.• Designed and developed an interactive Power BI dashboard for tracking and analyzing online sales data, incorporating complex drill-down functionalities and customized filters.• Integrated new tables, performed data joins, and calculated metrics for visualization purposes, utilizing various visualizations such as bar charts, pie charts, and donut charts.• Collaborated with stakeholders to refine the dashboard iteratively, ensuring it met evolving business needs while ensuring scalability and performance optimization for handling large datasets efficiently.