Senior Data Engineer
Current- Led the development and maintenance of comprehensive ETL and data ingestion pipelines on AWS for Bemobi's solutions (end-to-end customer engagement platform (SaaS) in payment solutions for Telecom, Utilities, and other industries), significantly enhancing more than 100TB data processing efficiency and reliability across 50+ countries using Airflow and Spark.- Designed and implemented scalable data pipelines using Python, PySpark, and SQL, extracting data from diverse sources to a centralized data lake, enhancing data availability for analytics and business intelligence.- Spearheaded cost optimization and performance enhancement initiatives within AWS (ECS, EC2, EMR, Glue, Redshift, RDS, Lambda, S3), markedly improving data processing speed and reducing operational costs.- Acted as the lead engineer for the seamless migration of data pipelines from acquired companies, ensuring zero downtime, and maintaining data integrity. Successfully integrated more than 200GB/day pipelines within 10 months.- Developed and maintained CI/CD pipelines and using Terraform (IaC), improving deployment efficiency by 87,5% and reducing manual errors.- Collaborated with Data and Business areas to refine data tables and deliverables, enhancing data-driven decision-making. Providing data accessibility to non-technical partners of more than 20TB.- Led the initiative for real-time data processing and streaming, employing technologies like Kafka and pub-sub mechanisms, which enabled real-time analytics and supported business operations with timely insights.- Worked closely with cross-functional teams to translate business requirements into scalable data engineering solutions, directly contributing to acquired companies' projects, and enhancing customer data access through APIs and documentation.- Mentored junior data engineers, fostering a culture of technical excellence and continuous learning within the team, which led to enhanced productivity and streamlined onboarding processes.