-Migrated the data from traditional database systems to Azure SQL databases. -Design and implement end-to-end data solutions like storage, integration, processing, and visualization in Azure. -Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform, and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool, and backward. -Worked on implementing Extraction, Transformation Load, and data movement solutions using Azure Data Factory, and SSIS. -Analyzed implementation of ad-hoc solutions using Azure Data Lake Analytics/Store, HDInsight. -Developed Spark and Scala scripts for extracting data from various databases. -Worked on design and implementation of streaming solutions using Kafka or Azure Stream Analytics.-Managed the data in the Azure Data Lake Storage & Data Lake Analytics and an understanding of how to integrate with other Azure Services. -Migrating the Pig scripts and MapReduce programs to Spark SQL to improve the performance and moved the log files generated from various sources to HDFS and processed the files. -Worked on transforming the data flow and streaming the data-driven workflows using Apache Spark and Python. -Worked with different file formats and data platforms, specifically SQL Server and related technologies such as SSIS, and SSRS for integration and reporting the data. -Worked on developing the ETL pipelines in Spark applications using Python and used Airflow for creating workflows. -Recreated the logic & functionality of the existing application in Azure Data Lake, Data Factory, SQL Database & SQL Datawarehouse environment & worked on BI project implementation using Azure DF.Environment: Python, SQL, Scala, Spark, PySpark, HiveQL, Matillion, Azure Data Lake, DB2, Teradata, HDFS, Sqoop, Azure Data Factory, Blob storage, Databricks, Kafka, JSON, Parquet, PySpark, ETL, Azure SQL DB, Azure Event Hubs, GIT, Unix, Power BI, and Tableau

-Migrated the data gathered from the internal sources to AWS using the EC2 instances for processing the data and S3 buckets for storage. -Worked with AWS Glue to transform the data from the S3 buckets to the target database in Spark and configured the data load in the data pipeline from the S3 buckets into Redshift. -Worked with AWS Cloud Watch to collect the data and monitor instances. -Developed real-time data pipeline using Spark to ingest customer events/activity data into Hive and Cassandra from Kafka. -Created the Hive tables for further transforming the data into the HDFS for storing the data using the Pig scripts. -Worked on building the Spark applications for extracting and transforming data from the workflows using Scala. Used Spark SQL for Scala & a Python interface that automatically converts RDD case classes to schema RDD. -Monitoring the SQL scripts and improving the performance by modifying scripts using PySpark SQL.-Experience in Migrating SQL database to Azure Data Lake, Azure Data Lake Analytics, Azure SQL Database, Data Bricks, and Azure SQL Data Warehouse and controlling and granting database access and Migrating On-premises databases to Azure Data Lake store using Azure Data Factory. -Migrated applications from internal data storage to Azure and migrated the data from HiveQL to Azure and created Hive tables, loading, and analyzing data using Hive scripts. -Worked on writing the Databricks code and ADF pipeline fully parameterized for efficient code management. -Developed Data bricks Python notebooks to Join, filter, pre-aggregate, and process the files stored in Azure data lake storage.Environment: IBM DataStage, Python, Spark framework, AWS, Redshift, MS Excel, NoSQL, Tableau, T-SQL, ETL, RNN, LSTM MS Access, XML, MS Office 2007, Outlook, MS SQL Server, Azure Data Lake, Azure Data Factory, Azure SQL DB, Azure Event Hubs

-Worked on Distributed/Cloud Computing (Map Reduce/Hadoop, Hive, Pig, HBase, SQOOP, Flume, Spark, AVRO, Zookeeper, etc.)-Migrated an existing on-premises application to AWS using AWS services like EC2 and S3 for processing and storage of data sets. -AWS Glue for transforming and validating the data extracted from the databases. -Worked on AWS Step Functions to automate and orchestrate the Amazon Sage Maker tasks such as publishing data to S3, training the ML model, and deploying it for prediction. -Worked on implementing AWS services - Elastic Compute Cloud (EC2), Lambda, Simple Storage Service (S3), Auto Scaling, AWS Glue, AWS Batch, DynamoDB, IAM, Virtual Private Cloud (VPC), Route53, Cloud Watch, AWS CLI, Cloud Formation, ELB (Elastic Load Balancers), RDS, SNS, SQS, and EBS, etc.,Extensive experience with DBT (Data Build Tool) for transforming and modeling data, optimizing ELT processes within GCP environments.Built and optimized data storage solutions on GCP using BigQuery, Cloud Storage, and Cloud SQL, enhancing data retrieval, scalability, and cost efficiency.Led the migration of data pipelines from traditional RDBMS and Big Data platforms to GCP, ensuring minimal downtime and data integrity.Implemented streaming solutions using Google Pub/Sub and Kafka, facilitating real-time data ingestion and processing.Optimized data processing workflows using GCP services like Dataflow, reducing costs and increasing efficiency while ensuring scalability.Hands-on experience with NoSQL databases, such as Bigtable and HBase, for managing unstructured and semi-structured data in cloud environments.Environment: AWS EC2, S3, Lambda functions, RDS, Redshift, Cloud Watch, Glue, Athena, Python, HDFS, SQL, Hive, PySpark, Spark, Snowflake, ETL, Scala, Spark-SQL, HBase, Apache Airflow, Shell, NoSQL, Cassandra, Kafka, YARN, Hive, MapReduce, and Kafka.

-Designed automation solutions for medium and high-complexity processes and worked around front-end applications via bots using Automation Anywhere (V11, A360), UiPath, and Blue Prism. -Implemented, modified, and optimized complex queries to data retrieval using SQL/Relational database. -Used various Excel actions such as Formulas (V-lookup, for, if, nested if, etc.), pivot table to tally the correct output, and Formatting to perform Excel automation. -Created bots to automate processes which saved manual efforts and time by 75-80% a week and improved the output by 30-40% due to less human error. -Used different SQL servers like SSMS and Oracle to perform operations/modifications on Excel data or extracted data from web applications. -Established quality and scalable programs, including analyzing, reporting, and problem-solving with network administrators, systems analysts, and other software developers to assist in resolving problems, and leadership knowledge. -Experienced in developing and executing change management plans, communication strategies, and training programs.