• Worked with data transfer from on-premisesSQL servers to cloud databases (Azure Synapse Analytics (DW) and Azure SQL DB). • Demonstrated expert level technical capabilities in areas of Azure Batch and Interactive solutions, Azure Machine learning solutions.• Created and Managed Storage accounts in Azure Portal, and pipelines, links, and datasets in Azure Data Factory (ADF).• Provide advice and cloud architecture for top clients and internal projects running on the Microsoft Azure platform to ensure high service availability and minimal operational expenses.• Experience managing Azure Data Lakes (ADLS) and Data Lake Analytics and an understanding of how to integrate with other Azure Services.• Moderate and contribute to Microsoft Developers Network support forums (special to Azure Networking, Azure Virtual Machines, Azure Active Directory, and Azure Storage) for Partners and MVPs.• Monitored the SQL scripts and modified them for improved performance using PySpark SQL. • Managed relational database service in which Azure SQL manages scalability, stability, and maintenance. • Integrated data storage options with Spark, notably with Azure Data Lake Storage and Blob storage. • Configured stream analytics, Event Hubs, and worked with Azure to manage IoT solutions.• Used Kubernetes for the runtime environment of the CI/CD system to build, test deploy.• Developed Json Scripts for deploying the Pipeline in Azure Data Factory (ADF) that process the data using the Cosmos Activity.• Implemented data ingestion from various source systems using Python and Pyspark. • Used Kubernetes to manage Docker orchestration and containerization. • Used Kubernetes to orchestrate the deployment, scaling, and management of Docker Containers.

• Involved in complete Big Data flow of the application data ingestion from upstream to HDFS, processing the data in HDFS and analyzing the data using several tools. • Creating Storage Pool and Stripping of Disk for Azure Virtual Machines. Backup, Configure and Restore Azure Virtual Machine using Azure Backup.• Worked on Databricks for ETL Data Cleansing, Integration & Transformation using python scripts for managing data from disparate sources. • Setup Databricks Enterprise Platform environment created cross-account role in AWS for Databricks to provision Spark clusters.• Using terraform the ability to enforce data security and compliance requirements have been made.• Imported the data from various formats like JSON, ORC and Parquet to HDFS cluster with compressed for optimization. • Experienced on ingesting data from RDBMS sources like - Oracle, SQL Server and Teradata into HDFS using Sqoop. • Importing and exporting data into HDFS and hive using Sqoop and Kafka with batch and streaming. • Experienced with Spark-Streaming APIs to perform transformations and actions on the fly for building the common learner data model which gets the data from Kafka in near real time and persists into HBase. • Workflow management and data pipeline experience with Airflow and knowledge of creating standards for DAGs, tasks, and airflow clusters, etc.

• Importing and exporting data into HDFS and Hive using SQOOP. • Installed Hadoop, Map Reduce, HDFS, and Developed multiple MapReduce jobs in PIG and Hive for data cleaning and pre-processing. • Data Extraction, aggregations, and consolidation of Adobe data within AWS Glue using PySpark.• Used Pig as ETL tool to do Transformations, even joins and some pre-aggregations before storing the data onto HDFS. • Experience working in spark eco system using Spark-Sql and Scala queries on different formats like Text file, CSV file. • Responsible for designing data pipelines using ETL for effective data ingestion from existing data management platforms to enterprise Data Lake.• Designed and developed MapReduce jobs to analyze the data and implemented best income logic using Pig scripts and UDFs. • Worked with application teams to install operating system, Hadoop updates, patches, version upgrades as required. • Responsible to manage data coming from different sources

• Developed Spark/Scala, Python for regular expression (regex) project in the Hadoop/Hive environment with Linux/Windows for big data resources. • Experienced in handling large datasets using Partitions, Spark in Memory capabilities, Broadcasts in Spark, Effective & efficient Joins, Transformations and other during ingestion process itself. • Designed, developed, and did maintenance of data integration programs in a Hadoop and RDBMS environment with both traditional and non-traditional source systems as well as RDBMS and NoSQL data stores for data access and analysis. • Analyzed the SQL scripts and designed the solution to implement using spark. • Responsible for developing data pipelines with Amazon AWS to extract the data from weblogs and store in HDFS. • Involved in creating Hive tables and loading and analyzing data using hive queries.