1. Optimized pipeline execution runtimes by application-level caching strategies, managing memory usage and graceful cache clearing.2. Developed licensing features to manage, limit, and monitor the usage of internal services, incorporating observability metrics and creating admin dashboards for easy monitoring and control.3. Implemented migration strategies to transition multiple applications and user-specific assets to support critical version updates, without disrupting user experience.

1. Designed and developed a web service using Django Rest Framework to manage the flow of news articles through multiple machine learning services deployed on AWS.2. Implemented asynchronous data pipelines using Celery, with RabbitMQ as the message broker, to dynamically manage worker processes. Added application-level caching layer with Redis and integrated Elasticsearch for faster searching across MongoDB documents.
3. Set up docker-compose to manage the deployment of multiple containers for the Django apps, RabbitMQ, and MongoDB on AWS EC2 instances.4. Set up and deployed Airflow Directed Acyclic Graphs (DAGs) to schedule ETL pipelines handling data transfer and transformations across multiple Amazon S3 buckets, Azure Blob Storage, and Snowflake databases.5. Worked on credit risk models, performing data engineering on large financial datasets with over 200 million rows using PySpark on Azure Databricks.