• Worked with Hadoop ecosystem and Implemented Spark using Python and utilized Data frames and Spark SQL API for faster processing of data.• Developed Spark Streaming job to consume the data from the Kafka topic of different source systems and push the data into HDFS locations.• Data sources are extracted, transformed, and loaded to generate CSV data files with Python programming and SQL queries.• Developed Automation Regressing Scripts for validation of ETL process between multiple databases like AWS Redshift, Postgres, MongoDB, T-SQL, and SQL Server using Python. • Used Airflow for scheduling the Hive, Spark, and MapReduce jobs.• Developed and managed ETL workflows on Snowflake using AWS infrastructure, automating data integration from diverse sources while ensuring data integrity and security through IAM roles and encryption mechanisms.• Converting Hive/SQL queries into Spark transformations using Spark RDDs and Pyspark• Filtering and cleaning data using Python code and SQL Queries• Created Snow pipe for continuous data load from staged data residing on cloud gateway servers.• Troubleshooting errors in Hbase Shell/API, Pig, Hive and MapReduce.• Implemented Installation and configuration of multi-node cluster on Cloud using Amazon Web Services (AWS) on EC2.

• Took projects ideas from business, lead, developed and created production pipelines that deliver business value using Azure Data Factory• Integrated Databricks with Azure Data Lake Storage for scalable data storage and retrieval, facilitating advanced analytics and machine learning projects.• Installed Kafka Producer on different severs and scheduled to produce data for every 10 seconds Implemented Data Quality in ETL Tool Talend • Having good knowledge in Data Warehousing Developed Apache Spark applications by using spark for data processing from various streaming sources.• Build the Logical and Physical data model for snowflake as per the changes required.• Implement Continuous integration/continuous development best practice using Azure DevOps, ensuring code versioning.• Developed HIVE UDFs to incorporate external business logic into Hive script and developed join data set scripts using HIVE join operations.• Optimized Snowflake queries and data models to improve processing efficiency and reduce costs.• Good Exposure on Map Reduce programming using Java, PIG Latin Scripting and Distributed Application and HDFS.

• Experience in configuring the Zookeeper to coordinate the servers in clusters and to maintain the data consistency which is important for decision making in the process.• Worked on different data formats such as JSON, XML and performed machine learning algorithms in Python.• Application development using Hadoop Ecosystems such as Spark, Kafka, HDFS, HIVE, Oozie and Sqoop.• Involved in support for Amazon AWS and RDS to host static/media files and the database into Amazon Cloud.• Automated RabbitMQ cluster installations and configuration using Python/Bash.• Designed and Developed Real Time Data Ingestion frameworks to fetch data from Kafka to Hadoop.• Using partitioning and bucketing in HIVE to optimize queries.• Used Python Library Beautiful Soup for web scrapping to extract data for building graphs.• Continuous monitoring and managing the Hadoop cluster through Cloudera Manager.

• Worked with Informatica Power Center for data processing and loading files.• Extensively worked with Informatica transformations.• Worked with SQL*Loader tool to load the bulk data into Database.• Developed mappings using Informatica to load data from sources such as Relational tables, Sequential files into the target system.• Created data maps in Informatica to extract data from Sequential files.• Utilised ER Diagrams to clearly Visualise the data relationships, ensuring accurate database design and effective communication with stakeholders.