Data Engineer
CurrentAchievements:- Reduced costs by 75% by developing a new ETL and data ingestion pipeline on AWS, including data quality.- Delivered reports 32x faster by implementing an automated tool that generates custom reports. - Implemented a pipeline for automatic inclusion of analysis notebooks into a portal with authentication, connected to the Jira API.- Developed a Python package to be used as a tool kit for analysis.Context:Work at Bemobi, which provides an end-to-end customer engagement platform (SaaS) in payment solutions for Telecom, Utilities, and other industries, as a member of the fraud-prevention team, responsible for maintaining the ETL and data ingestion process, and developing tools used by data analysts.Technologies:- Apache Airflow- Python- SQL- Apache Spark- AWS (EC2, S3, SES, Lambda, EC2, Fargate, Cognito)- GCP- DockerActivities:• Develop and maintain a comprehensive ETL and data ingestion pipelines on AWS, by using Airflow and Spark, for Bemobi's fraud prevention team, efficiently processing over half a million transactions daily with opsgenie integration for alert management.• Developed and implemented an automated reporting solution that schedules, generates, processes, and delivers customized reports, which are also available on a centralized page with user access control.• Developed and maintained CI/CD pipelines using Terraform (IaC)• Maintains a custom docker image used in our analyses• Developed a Private Python package, completely integrated into our custom docker image and pipelines.• Responsible for the integration of new products into our fraud platform