• Worked on revampification and optimization of the existing bigdata pipelines. • Worked on data pipelines orchestration tools like (Oozie) and Cloud Services (AWS). • Worked on big data technologies (Hadoop, Spark, Hive, Pig, HBase, Presto) to process large datasets. • We have Implemented containerization using Docker and orchestration using Kubernetes technologies. • Created and maintained documentation for data pipelines and architecture, including technical documents, user guides, and troubleshooting guides.• Mentored junior engineers to deliver data-driven insights and solutions to the organization. • Collaborated with cross-functional teams, including data scientists and software engineers, to deliver data-driven solutions.

• Assisted in the design and implementation of data pipelines and ETL processes.• Designed, built and maintained real-time data pipeline tool to monitor batch jobs and to Ingest millions of records from structured and semi-structured data to support data-driven decision-making using Scala, Kafka and Spark.• Worked on data modeling and data governance to ensure data quality• Review performance stats and query execution/explain plans and recommend changes for tuning Hive/Impala queries• Installation and configuration of Cloudera cluster and deployment of services (HDFS,HBASE,HIVE,SPARK,etc) and services high availability and maintenance of cluster.• Implement security measures for all aspects of the cluster (SSL, disk encryption, role-based access).• Performance tuning of Hadoop clusters and monitoring of job performances and analysis of Hadoop job logs.• Develop and automate processes for maintenance of the site by shell script.• Collaborating with application teams to install Hadoop updates, patches, version upgrades and Addition or removal of nodes in cluster.