Senior Data Engineer
CurrentLed data migration from Oracle and SAP ASE to BigQuery using Dataflow, staging in Google Cloud Storage (GCS).Containerized Python-based data processing jobs using Google Kubernetes Engine (GKE).Automated batch processing and task scheduling with Airflow for orchestration.Built real-time data pipelines from SAP HANA to Google Cloud for faster analytics using SAP BODS.Cleaned and normalized datasets using Spark and Python to get rid of inconsistencies.Scaled data pipelines to process huge amount of data daily to optimize performance and better latency.Implemented Change Data Capture (CDC) and incremental loading strategies for real-time data updates.Managed data ingestion pipelines with Terraform and set up CI/CD pipelines using Cloud Build.Monitored data workflows with Cloud Monitoring to ensure stability across BigQuery, Dataproc, and Airflow environments.Collaborated with cross-functional teams in an Agile environment, ensuring timely delivery of data solutions.