Karthikeyan R Email & Phone Number

🚀 Experienced Data EngineerI bring a wealth of experience as a Data Engineer specializing in cloud migration. I played a pivotal role in the Migration Team, focusing on the seamless transition from on-premises Cloudera to Google Cloud Platform (GCP) and optimizing data pipelines for maximum efficiency.🔍 Key Accomplishments:Migration Expertise: Spearheaded the migration efforts, orchestrating the movement of data from diverse sources, including Oracle, MS SQL databases, S3, FTP, and APIs, to BigQuery on GCP.Data Ingestion Mastery: Extensively designed and implemented data pipelines using Cloud Data Fusion and orchestrated them with Directed Acyclic Graphs (DAGs) in Cloud Composer. Primarily focused on migrating data from diverse sources like Oracle, MS SQL databases, S3, FTP, and APIs to BigQuery.🛠️ Technical Proficiency:Cloud Technologies: Proficient in GCP with a strong emphasis on BigQuery, Cloud Data Fusion, Cloud Dataproc and Cloud Composer (Airflow.Pyspark Wizardry: Developed frameworks using Pyspark for ingesting data from file-based systems (FTP, S3, API) to BigQuery on Cloud Dataproc, supporting multiple file formats such as XML, JSON, CSV, XLSX, and Parquet.Certifications: Certified in GCP, AWS, and Azure, demonstrating a broad and versatile skill set across major cloud platforms and recognized as a Databricks Certified Spark Developer, showcasing proficiency in Spark development.Incident Handling: Analytically troubleshooted failure incidents, providing timely and effective solutions. Implemented email notifications using the SMTP Python module for ingestion checkpoints and data quality rule checks.🌐 Optimization Guru:Cost Savings: Contributed significantly to the Optimization Team by scaling down worker nodes and configuring Pyspark jobs to execute on create-use-delete Spark clusters, resulting in cost savings of 31%-42%.BigQuery Optimization: Expertise in partitioning and clustering BigQuery tables, identifying cluster usage patterns for high-load intensive jobs.📧 Communication and Collaboration:Email Notification: Configured email notifications using smtp servers for ingestion checkpoints ,data quality rule checks and on successful data ingestion also configured task-level email notifications for failure in DAGs , ensuring a proactive approach to issue resolution.