• Extensively used AWS Athena to import structured data from S3 into other systems such as RedShift to generate reports.• Worked with Spark to improve the speed and optimization of Hadoop's current algorithms.• Created a Data Pipeline utilizing Processor Groups and numerous processors in Apache Nifi for Flat File, RDBMS as part of a Proof of Concept (POC) on Amazon EC2.• Migrated an existing on-premises application to AWS. AWS services such as EC2 and S3 were used for data set processing and storage. Experienced in maintaining a Hadoop cluster on AWS EMR.• Created Apache presto and Apache drill configurations on an AWS EMR (Elastic Map Reduce) cluster to integrate different databases such as MySQL and Hive. This allows for the comparison of outcomes such as joins and inserts on many data sources controlled by a single platform.• Performed end-to-end architecture and implementation evaluations of different AWS services such as Amazon EMR, Redshift, S3, Athena, Glue, and Kinesis. Hive. Created external table schemas for the data being processed as the primary query engine of EMR.• AWS RDS (Relational database services) was created to act as a Hive meta store, and metadata from 20 EMR clusters could be integrated into a single RDS, preventing data loss even if the EMR was terminated.• Developed and implemented ETL pipelines on S3 parquet files in a data lake using AWS Glue.• Developed a cloud formation template in JSON format to utilize content delivery with cross-region replication using Amazon Virtual Private Cloud.• The AWS Code Commit Repository was utilized to preserve programming logic and scripts, which were subsequently replicated to new clusters.

• Developed end-to-end data pipelines using Azure Data Factory, enabling efficient data ingestion, transformation, and loading processes.• Designed and implemented scalable and high-performance data processing solutions using Azure Databricks, leveraging Spark-based technologies for large-scale data analytics and machine learning.• Built data warehousing solutions on Azure Synapse Analytics, including data modeling, partitioning strategies, and performance optimization for complex analytical queries.• Implemented secure data storage and access controls using Azure Data Lake Storage and Azure Blob Storage, ensuring data integrity and compliance with privacy regulations.• Leveraged Azure SQL Database and other Azure database services to design and optimize database schemas, improve query performance, and ensure data consistency and reliability.• Implemented data governance practices, including data lineage, metadata management, and data quality validation, to ensure data integrity and facilitate compliance.• Conducted performance tuning and optimization for Azure data solutions, improving data processing efficiency, reducing costs, and enhancing overall system performance.• Collaborated closely with cross-functional teams, including data scientists, business analysts, and stakeholders, to understand requirements, define data engineering solutions, and deliver successful projects.• Led the implementation of monitoring and alerting mechanisms, utilizing Azure Monitor and other tools, to proactively identify and address issues, ensuring the reliability and availability of data solutions.• Demonstrated strong problem-solving skills by troubleshooting and resolving complex data engineering challenges, ensuring data accuracy, and meeting project deadlines.• Kept abreast of the latest Azure data technologies, trends, and best practices, proactively applying them to enhance data engineering processes and drive continuous improvement.

• Extensively used Informatica Client tools Power Center Designer, Workflow Manager, Workflow Monitor and Repository Manager. Extracted data from various heterogeneous sources like Oracle, Flat Files.• Performed data operations like Text Analytics and Data Processing, using the in-memory computing capabilities of Spark using Scala. • Implement Data Interface to get information on customers using Rest API.• Maintained ELK (Elastic Search, Logstash, and Kibana) and Wrote Spark scripts using Scala. • Designed SSIS (ETL) Packages to extract data from various heterogeneous data sources such as Access database, Excel spreadsheet and flat files into SQL Server and maintain the data.• Creating Informatica workflows to load the source data into CSDR. Involved in creating various UNIX script used during ETL load process.• Had experience in Hadoop framework, HDFS, MapReduce processing implementation. • Worked on the development of data warehouse, Data Lake and ETL systems using relational and non-relational tools like SQL, No SQL. In in converting the business requirement into technical design document.• Creating new repositories from scratch, backup and restore• Used Spark-SQL to read, process the parquet data, and create the tables using the Scala API. • Monitored Spark Application to capture the logs generated by Spark jobs.• Implement MapReduce using Sqoop to import the data from various relational databases to filter out unstructured data, perform data cleaning and pre-process on HDFS.• Periodically cleaning up Informatica repositories. Monitoring the daily load & handing over the stats with the QA Team.