🚀 Experienced Data EngineerI bring a wealth of experience as a Data Engineer specializing in cloud migration. I played a pivotal role in the Migration Team, focusing on the seamless transition from on-premises Cloudera to Google Cloud Platform (GCP) and optimizing data pipelines for maximum efficiency.🔍 Key Accomplishments:Migration Expertise: Spearheaded the migration efforts, orchestrating the movement of data from diverse sources, including Oracle, MS SQL databases, S3,… Show more 🚀 Experienced Data EngineerI bring a wealth of experience as a Data Engineer specializing in cloud migration. I played a pivotal role in the Migration Team, focusing on the seamless transition from on-premises Cloudera to Google Cloud Platform (GCP) and optimizing data pipelines for maximum efficiency.🔍 Key Accomplishments:Migration Expertise: Spearheaded the migration efforts, orchestrating the movement of data from diverse sources, including Oracle, MS SQL databases, S3, FTP, and APIs, to BigQuery on GCP.Data Ingestion Mastery: Extensively designed and implemented data pipelines using Cloud Data Fusion and orchestrated them with Directed Acyclic Graphs (DAGs) in Cloud Composer.🛠️ Technical Proficiency:Cloud Technologies: Proficient in GCP with a strong emphasis on BigQuery, Cloud Data Fusion, Cloud Dataproc and Cloud Composer (Airflow.Pyspark Wizardry: Developed frameworks using Pyspark for ingesting data from file-based systems (FTP, S3, API) to BigQuery on Cloud Dataproc, supporting multiple file formats such as XML, JSON, CSV, XLSX, and Parquet.Incident Handling: Analytically troubleshooted failure incidents, providing timely and effective solutions. Implemented email notifications using the SMTP Python module for ingestion checkpoints and data quality rule checks.🌐 Optimization Guru:Cost Savings: Contributed significantly to the Optimization Team by scaling down worker nodes and configuring Pyspark jobs to execute on create-use-delete Spark clusters, resulting in cost savings of 31%-42%.BigQuery Optimization: Expertise in partitioning and clustering BigQuery tables, identifying cluster usage patterns for high-load intensive jobs.📧 Communication and Collaboration:Email Notification: Configured email notifications using smtp servers for ingestion checkpoints ,data quality rule checks and on successful data ingestion also configured task-level email notifications for failure in DAGs , ensuring a proactive approach to issue resolution. Show less

GCP Data Engineer having exposure to Bigquery, Cloud Dataproc, Datafusion, Airflow, PubSub, Cloud Function, GCS and Bigquery data transfer services to migrate on-prem tech to Cloud