Working as data engineer for building scalable distributed data solutions using Hadoop.Installed and configured Hive, Sqoop and Flume on the Hadoop cluster.Developed Business logic in semantic layer by creating views in AWS Redshift to provide transformation logic visibility. Developed PySpark and Spark SQL code to process the data in Apache Spark on Amazon EMR to perform the necessary transformations based on the STMs developed.

Worked with data governance team on data protection regulation projects.Developed Big Data solutions focused on pattern matching and predictive modeling. Extracted the data required for building models from AWS snowflake Database. Pulling the data from data lake (HDFS) and massaging the data with various RDD transformations.

Worked with the analysis teams and management teams and supported them based on their requirements. Collaborate with application architects and DevOps. Designed and Configured Azure Cloud relational servers and databases analyzing current and future business requirements. Worked on migration of data from On-prem SQL server to Cloud databases (Azure Synapse Analytics (DW) & Azure SQL DB). Played a key role in migrating Azure SQL, Hadoop cluster on MS Azure and defined different read/write strategies. Developed live reports in a drill down mode to facilitate usability and enhance user interaction.

Developed stored procedures, functions, views and triggers in PL/SQL to be used in the ETL process.Utilized Sqoop, Kafka, Flume and Hadoop File System API's for implementing data ingestion pipelines.Worked on real time streaming, performed transformations on the data using Kafka and Spark Streaming. Created Hive tables, loaded with data, and wrote Hive queries to process the data.Worked with Business users and Business analysts to understand and analyze ETL and reporting requirements.

Worked on Performance Tuning of the database which includes indexes, optimizing SQL Statements.Coordinated with the Business Analyst and prepared Logical and Physical Data models as per the requirements involving multiple subject areas, domains and naming standards.Developed data Mart for the base data in Star Schema, Snowflake Schema involved in developing the data warehouse for the database.Prepared complex T-SQL queries, views and stored procedures to load data into staging area.