•Led a team of engineers in designing and implementing a scalable and high-performance Databricks architecture, ensuring optimal resource utilization and cost efficiency.•Led the design and implementation of Kafka-based data streaming pipelines, ensuring high throughput, fault tolerance, and low latency for critical business processes.•Orchestrated the migration from ZooKeeper-based Kafka clusters to ZooKeeper-less Kafka clusters, resulting in improved cluster stability and easier maintenance.•Collaborated with cross-functional teams to develop custom Kafka Connect connectors for integrating Kafka with various external systems, reducing data integration complexities.•Managed the Schema Registry to enforce schema compatibility, versioning, and data governance standards across the organization's Kafka topics.•Leveraged KSQL to build real-time stream processing applications, enabling the extraction of actionable insights from streaming data.•Configured and optimized Rest Proxy for secure and efficient RESTful access to Kafka topics, enhancing data accessibility for external applications.•Implemented Mirror Maker to replicate Kafka data between geographically distributed data centers, ensuring data redundancy and disaster recovery capabilities

• Utilized Spark SQL API in PySpark to extract and load data and perform SQL queries.• Worked on developing PySpark script to encrypting the raw data by using hashing algorithms concepts on client specified columns.• Extracted, transformed, and loaded data from various source systems into Azure Data Storage services using Azure Data Factory, T-SQL, Spark SQL, and U-SQL Azure Data Lake Analytics.• Processed data in Azure Databricks and ingested it into Azure services such as Azure Data Lake, Azure Storage, Azure SQL, and Azure Data Warehouse.• Created pipelines in Azure Data Factory using Linked Services, Datasets, and Pipelines to extract, transform, and load data from different sources like Azure SQL, Blob storage, Azure SQL Data Warehouse, and write-back tools.• Developed PySpark and Spark-SQL applications for data extraction, transformation, and aggregation from multiple file formats, providing insights into customer usage patterns.• Assumed responsibility for estimating cluster size, monitoring, and troubleshooting Spark Databricks clusters.• Proficient in optimizing the performance of Spark applications, including batch interval time, parallelism level, and memory tuning.• Responsible for Design, Development, and testing of the database and Developed Stored Procedures and Views.

• Developed Spark applications using PySpark and Spark SQL to extract, transform, and aggregate data from multiple file formats, enabling analysis and transformation to uncover insights into customer usage patterns.• Led the architecture and design of a large-scale data warehousing system, incorporating best practices in data modeling and dimensional modeling.• Worked closely with business stakeholders to define data architecture strategies that aligned with organizational goals.• Ensured data governance and quality by implementing data profiling, data cleansing, and data lineage tracking.• Collaborated with AWS cloud architects to leverage DynamoDB and other AWS services for high-performance NoSQL database solutions.• Developed cost optimization strategies, leading to a 15% reduction in operational expenses.• Mentored junior data engineers and provided technical guidance on data modeling, architecture, and database optimization.• Possess a solid understanding of Spark Architecture, including Spark Core, Spark SQL, Data Frames, Spark Streaming, driver node, worker node, stages, executors, and tasks.• Wrote Spark jobs using Scala to interact with PostgreSQL databases through Spark SQL Context and accessed Hive tables using Hive Context.