As a Data Engineer specializing in Databricks, I have achieved certifications for both Data Engineer Associate and Professional, showcasing my expertise in data engineering practices and tools. I played a key role in a large-scale migration project, successfully transpiling Snowflake SQL code to Databricks SQL and migrating approximately 5 petabytes of data across 9,000 tables. I developed and optimized Databricks workflows and scheduled jobs to streamline data processing, while also automating… Show more As a Data Engineer specializing in Databricks, I have achieved certifications for both Data Engineer Associate and Professional, showcasing my expertise in data engineering practices and tools. I played a key role in a large-scale migration project, successfully transpiling Snowflake SQL code to Databricks SQL and migrating approximately 5 petabytes of data across 9,000 tables. I developed and optimized Databricks workflows and scheduled jobs to streamline data processing, while also automating the testing process using PySpark to ensure data integrity and accuracy throughout the migration. Additionally, I utilized Databricks APIs to automate workflow triggers and gather essential statistics for comprehensive reporting on the migrated data, including monitoring upstream and downstream tables, workflow run times, and identifying task-level errors. Show less

As a Data Engineer for Jio's PaaS product, Jio CloudXP, I contributed to the development of a full-stack hybrid and multi-cloud management platform designed to efficiently manage on-premises and cloud resources. I played a key role in the research and development of two major modules, focusing on cost management and observability. My responsibilities included implementing performance and memory optimizations for various Spark applications, which significantly enhanced overall efficiency and… Show more As a Data Engineer for Jio's PaaS product, Jio CloudXP, I contributed to the development of a full-stack hybrid and multi-cloud management platform designed to efficiently manage on-premises and cloud resources. I played a key role in the research and development of two major modules, focusing on cost management and observability. My responsibilities included implementing performance and memory optimizations for various Spark applications, which significantly enhanced overall efficiency and scalability. I conducted thorough analyses to identify bottlenecks and applied targeted optimizations to improve processing speed and reduce memory footprint. Additionally, I designed and executed a robust data processing framework for the Hybrid Cloud Management Platform (HCMP) project, utilizing Apache Kafka to ingest telemetry data from diverse PaaS services across Azure, GCP, AWS, and on-premises servers. I developed multiple Spark Streaming and Spark Structured Streaming applications to enable real-time processing and analysis of telemetry data, thereby contributing to enhanced observability within the platform. Show less

I designed and implemented the Generic Processing Utility, a robust ETL/ELT builder tool that leverages Apache Spark and Scala to facilitate the creation of code-free ETL/ELT pipelines for various batch and stream data integration and warehousing requirements. This tool significantly accelerates the development lifecycle by simplifying the process of building data pipelines. Additionally, I designed an intuitive user interface for the tool, enabling users to easily drag and drop components of… Show more I designed and implemented the Generic Processing Utility, a robust ETL/ELT builder tool that leverages Apache Spark and Scala to facilitate the creation of code-free ETL/ELT pipelines for various batch and stream data integration and warehousing requirements. This tool significantly accelerates the development lifecycle by simplifying the process of building data pipelines. Additionally, I designed an intuitive user interface for the tool, enabling users to easily drag and drop components of ETL/ELT pipelines and connect them based on their dependencies. This user-friendly approach empowers users to create complex data workflows without requiring extensive coding knowledge. Show less