Software Engineer
Current- Developed and designed a data platform from scratch processing large volumes of data using Python, SQL - Optimized data pipelines decreasing the execution time by 70% utilizing parallel processing of data partitions - Collaborated with Data Scientists & Analysts to design optimal data model and complex business logics - Orchestrated the ETL pipelines using Prefect to maintain and monitoring efficiently - Worked with git & CI/CD pipelines to automatically test & deploy the ETL jobs using Github Actions, Docker, AWS S3, ECS & ECR