Data Engineer
Developed and maintained ETL (Extract, Transform, Load) jobs using Apache Spark and Apache Airflow, ensuring efficient data processing and integration.Orchestrated automated application deployment pipelines using Gitlab CI/CD and Ansible, improving the efficiency and reliability of the deployment process.Designed, implemented, and tested a RESTful API using FastAPI, enabling efficient communication and data exchange between different systems.Collaborated with cross-functional teams to optimize AWS services for cloud-based solutions, enhancing scalability and performance.Led migration effort of Airflow jobs from EC2 to MWAA, including updating and refactoring code, contributing documentation, and enabling and monitoring (CPU and memory) alerts.