Senior Data Engineer
Current- Configured the GCP Cloud Build CI/CD to deploy the Airflow DAGS in Astronomer. Optimized ML Pipeline by segmenting the code into multiple jobs and executing these jobs concurrently on a Dataproc single cluster. This approach significantly reduced costs to 1/6 of the original and decreased the time complexity.- Configured the Cloud Function. Identified the pattern in complex Semi Structured Data (JSON file) and transformed the JSON into BQ loadable format and landed the data in BQ successfully using Cloud Function.- Configured Cloud Run and Cloud Scheduler services to automate the Python codes.