Cloud Data Engineer
Current• Migrated an existing Apache Airflow instance to a Dockerized environment, ensuring scalability and streamlined deployments• Designed and implemented Airflow DAGs to perform ETL processes, seamlessly transferring data from on-premise MS SQL Server to Azure SQL Databases.• Developed and maintained the infrastructure supporting Airflow, integrating Azure DevOps pipelines for CI/CD and managing Docker containers for deployment and orchestration.• Configured Datadog monitoring within the Docker environment to provide real-time insights and health monitoring of Airflow containers.