• Provided a solution using HIVE, and SQOOP (to export/ import data), for faster data load by replacing the traditional ETL process with HDFS for loading data to target tables. • Develop framework for converting existing PowerCenter mappings and to PySpark (Python and Spark) Jobs. • Developed complex data cleaning and transformation logic using PySpark on AWS Glue to process unstructured data from S3 into analytics-ready datasets in Redshift.• Created serverless ETL workflows in a cloud platform using AWS Glue, Glue Data Catalog, S3, RDS, Cloud Watch, and Lambda.• Using Scala to compare the performance of Spark with Map Reduce, Hive• Developed the Pig UDFs to preprocess the data for analysis.• Implemented pipeline to load XML into HDFS using STORM & FLINK.• Used Pig Latin and Pyspark scripts to extract the data from the output files, process it, and load it into HDFS.• Worked on Creating Custom Datasets for downstream reporting.• Implemented partitioning, dynamic partitions, and bucketing in HIVE.• Used the messaging Framework Kafka.• Provide guidance to development team working on PySpark as ETL platform• Implemented configuration and optimization techniques for Redshift clusters to maximize data processing performance and streamline query execution, resulting in high-performance data analytics capabilities• Optimize the Pyspark jobs to run on Kubernetes Cluster for faster data processing.• Implemented AWS Athena for ad-hoc data analysis and querying on data stored in AWS S3• Used Kafka with a combination of Apache Storm, Hive for real-time analysis of streaming of data.• Utilized AWS CloudWatch to monitor and handle resources, configure alarms, and gather metrics• Configured Spark streaming to receive real-time data from the Kafka and store the stream data to HDFS.• Creating Databricks notebooks using SQL, Python • Used Data formats like ORC, Avro, Parquet.

• Developed Spark jobs using Scala for faster real-time analytics and used Spark SQL for querying • Developed storytelling dashboards in Tableau Desktop and published them on to Tableau Server which allowed end users to understand the data on the fly with the usage of quick filters for on-demand needed information.• Manage Azure Active Directory and create new groups for users• Created Azure Data Factory pipelines for applying transformations using Databricks Spark and then finally moved/loaded the transformed data into the Curated Data Model.• Configuration of NoSQL (Azure Cosmos DB) database in an application for storing and fetching client-related data.• Designed and Developed comprehensive plans on Azure Data Factory (ADF) for consuming data from different sources. • Created event-driven architectures with azure event grid using blob triggers and ADF. • Designed and handled complex data transformations, pipelines, and data flow using ADF and PySpark with Databricks. • Automated tasks using ADF's Tumbling window, Events, and Schedules using triggers. • Creating Databricks notebooks using PySpark, and Python.• Created several Databricks Spark tasks using Pyspark to do various tables-to-table operations. • Worked on data flow activity to conduct the data flow transformation. • Implemented Azure, self-hosted integration runtime in ADF. • Developed streaming pipelines using Apache Spark with Python. • Managing various access controls of AAD through Identity governance, B2B, and B2C.

• Designed and implemented data transfer from and to Hadoop and AWS.• Configured flume to ingest near real-time data from various sources to HTTP server which is a stage in Stream Sets Pipeline. • Analyzed and visualized near real-time data in Impala and Solr through Hue and Banana UI to prepare reports. • Deployed Spark/PySpark solution of ingesting of the data, transforming, wrangling, and applying business logic to the data from HDFS to SOLR.• Acted as a lead resource and built the entire Hadoop platform from scratch.• Extracted the needed data from the server into HDFS and bulk-loaded the cleaned data into Hbase.• Lead role in NoSQL column family design, client access software, HBase tuning; during migration from Oracle-based data stores.• Created pipelines to ingest Data from Blob Storage and Data Lake into Synapse Analytics• Worked on migrating data from On-Premises SQL servers to Azure Synapse Analytics and Azure SQL and moving of SSIS Packages into multiple domains of Azure .• Worked on transformation and modification of data flows , complex data and pipelines using Azure Data Factory with Databricks• Created and provisioned different Databricks clusters, notebooks, jobs, and autoscaling.• Created stored procedures in Azure SQL DW and SnowFlake• Created ETL pipelines in Python and PySpark to load data into Hive tables under Databricks

• Importing and exporting data into HDFS from Oracle Database and vice versa using Sqoop.• Installed and configured Hadoop MapReduce, HDFS, Developed multiple MapReduce• Designed and implemented MapReduce-based large-scale parallel relation-learning system• Set up and benchmarked Hadoop/HBase clusters for internal use.• Involved in the review of functional and non-functional requirements.• Extensive use of Expressions, Variables, and Row Count in SSIS packages• Created SSIS packages to pull data from SQL Server and exported to Excel Spreadsheets and vice versa.• Loading data from various sources like OLEDB, and flat files to SQL Server database Using SSIS Packages and creating data mappings to load the data from source to destination.• Wrote MapReduce job using Pig Latin. Involved in ETL, Data Integration, and Migration.• Creating Hive tables and working on them using Hive QL. Experienced in defining job flows.• Created batch jobs and configuration files to create automated processes using SSIS.• Designed and implemented data transfer from and to Hadoop and AWS.• Deploying and scheduling reports using SSRS to generate daily, weekly, monthly, and quarterly reports.• Involved in creating Hive tables, loading the data, and writing Hive queries that will run internally in a map-reduced way. Developed a custom File System plugin for Hadoop so it can access files on the Data Platform. • The custom File System plugin allows Hadoop MapReduce programs, HBase, Pig and Hive to work unmodified and access files directly.