• Engineered end-to-end data solutions, including analysis, design, data pipeline engineering, testing, deployment, and production support.• Developed and optimized Apache Spark applications in Scala for high-performance data processing.• Managed data processing pipelines on Amazon EMR, processing petabytes of data efficiently.• Designed real-time data processing using EMR's integration with Apache Kafka.• Orchestrated EMR workflows using Amazon Step Functions for fault-tolerant, reliable processing.

• Experienced with batch processing of data sources using Apache Spark, Elastic search, and Databricks.• Designed, developed, tested, and implemented Ab Initio ETL graphs against high-volume and complex data sources.• Designed and implemented big data solutions using Databricks, including data processing pipelines and machine learning models.• Utilized Databricks' Spark-based platform to process large data sets, including data from various sources such as Apache Kafka, Amazon S3, and Google Cloud Storage.

• Proficient in using various GCP services, including Compute Engine, Kubernetes Engine, Cloud Storage, Cloud SQL, and BigQuery.• Expertise in Implementing Security Measures: A Hadoop security administrator should possess hands-on experience in implementing and managing security measures such as authentication, authorization, encryption, auditing, and network security. They should be knowledgeable about the various security mechanisms available in Hadoop and understand how to configure and monitor them effectively.

• Involved in the software development lifecycle (SDLC) of tracking the requirements, gathering, analysis, detailed design, development, system testing and user acceptance testing.• Developed entire frontend and backend modules using Python on Django Web Framework. • Involved in designing user interactive web pages as the front-end part of the web application using various web technologies like HTML, Spark Script, Angular JS, jQuery, AJAX and implemented CSS for better appearance and feel.

• Worked on analyzing Hadoop cluster and different big data analytic tools including Pig, Hive and Sqoop.• Created POC on Hortonworks and suggested the best practice in terms HDP, HDF platform.• Experience in understanding the security requirements for Hadoop and integrating with Kerberos authentication infrastructure- KDC server setup, managing. Management and support of Hadoop Services including HDFS, Hive, Impala, and SPARK.• Installing, Upgrading and Managing Hadoop Cluster on Cloudera.