Arun J Email & Phone Number

Cloud Data Engineer & Architect with over 10 years of experience, specializing in building and optimizing data platforms using Azure, Databricks, Snowflake, KAFKA and DBT. Skilled in designing end-to-end data pipelines, processing petabyte-scale datasets, and managing real-time streaming and batch data. Adept at ensuring data governance and regulatory compliance using Unity Catalog.Azure:• Azure Data Engineering & Architecture: Proficient in Azure Data Factory, Azure Synapse Analytics, Azure Data Lake, Azure SQL, Azure Event Hub, and Azure Functions for scalable, secure, and performant data solutions. Expertise in ETL/ELT processes and real-time data streaming.• Azure Event Hub & Streaming Data: Skilled in implementing real-time streaming data pipelines withAzure Event Hub, ensuring low-latency and high-throughput data processing.• Azure Synapse & Data Warehousing: Expertise in using Azure Synapse Analytics for large-scale datawarehousing, processing, and integration with the broader Azure ecosystem.Databricks• Data Engineering & Apache Spark: Deep expertise in Databricks and PySpark for big data processing,batch and streaming optimizations, and data transformation workflows. Experience in Delta Lake for data integrity, upserts, and deduplication of large datasets..• Unity Catalog & Auto Loader: Proficient in using Unity Catalog for data governance and Databricks Auto Loader for efficient ingestion of real-time data streams.• Performance Optimization: Proven experience in optimizing Apache Spark workflows, improving system reliability and enhancing throughput using Spark SQL and DataFrame API for dynamic transformations.• CI/CD & DevOps Integration: Expertise in setting up CI/CD pipelines using Azure DevOps, ensuring robust deployment of Databricks workflows, and collaborating with cross-functional teams to streamline data engineering processes.Snowflake:• Snowflake Data Warehousing: Having knowledge in Snowflake architecture, configuration, andoptimization for high-performance data warehousing. Hands-on experience with Snowpipe, Streams,Materialized Views, and role-based access control (RBAC) for managing large datasets.• Performance Tuning & Query Optimization: Having knowledge in optimizing Snowflake queries, tuningperformance, and managing resources for scalable and efficient data processing• Data Integration & ETL: Skilled in building and maintaining ETL pipelines, ensuring seamless integration with Snowflake and other cloud platforms.

• 

  Teegala Krishna Reddy Engineering College

  Computer Science