• Designed and executed intricate data pipelines via AWS Glue for efficient transformation and loading of extensive data from diverse sources.• Automated ETL processes through maintenance and optimization of Glue jobs and crawlers, ensuring seamless data processing and analysis.• Designed and developed efficient stored procedures in AWS Redshift to automate complex data transformations, aggregations, and calculations.• Utilized SQL scripting within AWS Redshift's stored procedures to optimize query performance and minimize data movement.• Engineered Spark, Hive, Pig, Python, Impala, and HBase data pipelines for seamless customer data ingestion and processing.• Experience in creating fact and dimensional model in MS SQL server and Snowflake Database utilizing cloud base Matillion ETL tool.• Orchestrated Amazon EC2 instances creation, troubleshooting, and health monitoring, alongside other AWS services for multi-tier application deployment.• Provided Linux and Windows cloud instances support on AWS, configuring Elastic IP, Security Groups, and Virtual Private Cloud.• Part of team that created unified data lake in AWS S3 to ingest different source of data to create an unified data platform in Redshift with enterprise data catalog to publish data for various applications and set central access control for data governance.• Virtualized servers using Docker for testing and development environments, streamlining configuration through Docker containers.• Managed Docker clusters, integrating them with Amazon AWS/EC2 and Google's Kubernetes.• Utilized Jenkins to deploy CI/CD pipeline jobs for end-to-end automation, overseeing artifact management in Nexus repository.• Developed and implemented automated test scripts for ETL processes using Python, SQL• Performed comprehensive manual testing on ETL pipelines, ensuring data integrity and accuracy across multiple sources and destinations

• Worked on AWS services with focus on big data analytics, enterprise data warehouse and business intelligence solutions to ensure optimal architecture, scalability, flexibility• Used AWS services like EC2 and S3 for small data sets.• Designed AWS architecture, Cloud migration, AWS EMR, Dynamo DB, Redshift and event processing using lambda function.• Designed and implemented a wide range of ETL pipelines using Azure Databricks to extract, transform, and load data from various sources, including flat files, databases, and APIs.• Implemented Data Validation using MapReduce programs to remove un-necessary records before move data into Hive tables• Created Hive tables for loading and analyzing data and developed Hive queries to process data and generate data cubes for visualizing• Extracted data from HDFS using Hive, Presto and performed data analysis using Spark with Scala, PySpark and feature selection and created nonparametric models in Spark• Built NoSQL solution for non-structural data using AWS Dynamo DB services.• Worked on Snowflake environment to remove redundancy and load real time data from various data sources into HDFS using Kafka• Integrated Spark with Hadoop eco systems and stores the data Hadoop File System (HDFS)• Developed Spark/Scala, Python for regular expression (regex) project in the Hadoop/Hive environment with Linux/Windows for big data resources.• Designed the data aggregations on Hive for ETL processing on Amazon EMR to process data as per business requirement• Used AWS Glue catalog with crawler to get the data from S3 and perform SQL query operations.• Worked on data extraction, aggregations and consolidation of Adobe data within AWS Glue using PySpark.• Created external tables with partitions using AWS Athena and Redshift.

• Design tables and views structure for the application. Define DDL and accordingly build the table structure. Upload the table into related databases and upload it in respective project in GIT• Creating views, stored procedures, appropriate functions, to facilitate efficient data manipulation and data consistency• Writing procedures and functions using dynamic SQL and wrote SQL queries using joins, sub queries and correlated sub queries. Writing new stored procedures and tune them so that they perform well• Effectively worked on Data migration. Built command files for data migration, files to generate view and do bulk insert and bcp out, migrate data from legacy to the target databases• Rebuilding the indexes at regular intervals for better performance.• Implementation of SQL Logins, Roles and Authentication Modes as a part of Security Policies for various categories of users • Created custom Python scripts to conduct data analysis, transformation, and visualization tasks.• Utilized SSIS extensively for manual data transfer from DTS packages to SSIS packages.• Leveraged Snowflake's semi-structured data support for handling JSON and XML data formats.• Developed a comprehensive reporting environment utilizing data from multiple sources and creating analysis using SQL Server Reporting Services (SSRS).• Designed, deployed, and maintained of various SSRS Reports in SQL Server 2012.• Refactoring Stored procedures and user defined functions for building online and offline reports using SSRS Backing up, restoring system & other databases as per requirements, and scheduled those backups.