Performed end-to-end Architecture & implementation assessment of various GCP services like Google Dataproc, BigQuery, and Google Cloud Storage (GCS).Used Spark SQL for Scala and Python interfaces that automatically convert RDD case classes to schema RDDs.Developed Spark RDD and Spark DataFrame API for Distributed Data processing.Implemented Automic to schedule and automate complex ETL workflows, ensuring timely and accurate data processing across GCP services like Dataproc, and Cloud StorageUsed Google Dataproc to transform and move large amounts of data into and out of other GCP data stores and databases, such as Cloud Storage.Imported data from different sources like HDFS/HBase into Spark RDD and performed computations using PySpark/Scala Spark to generate the output response.Led the migration of a large-scale data processing pipeline from Apache Spark 2.4 to Spark 3.0, ensuring minimal downtime and seamless transition.Optimized performance by leveraging new features in Spark 3, such as Adaptive Query Execution (AQE) and Dynamic Partition Pruning, resulting in a 20% reduction in job execution time.Refactored codebase to accommodate changes in the Dataset API, built-in UDFs, and functions, ensuring compatibility and improved efficiency.Upgraded dependencies and ensured compatibility with Scala 2.12, addressing any issues related to library updates and deprecations.Conducted extensive testing and validation to ensure data integrity and performance benchmarks were met post-migration.Documented migration process and provided training sessions for the team to familiarize them with new features and best practices in Spark 3.Involved in using GCP services like BigQuery, Cloud Dataflow, and Google Cloud Storage for ETL jobs.Build data pipelines in Airflow in GCP for ETL related jobs using different Airflow operators.Transformed UPC Item IDs using Apache Spark and Scala, implementing efficient data processing pipelines to standardize and clean large datasets.

Performed end- to-end Architecture & implementation assessment of various AWS services like Amazon EMR, Redshift, S3. Implemented Security Framework to provide fine grained access to objects in AWS S3 using AWS Lambda, DynamoDB. Used Spark SQL for Scala amp, and Python interface that automatically converts RDD case classes to schema RDD. Developed Spark RDD and Spark data frame API for Distributed Data processing.Explored PySpark framework on AWS Databricks for improving the performance and optimization of the existing algorithms in Hadoop using PySpark core, Spark SQL and Spark streaming API.Worked on Kerberos authentication principals to establish secure network communication on cluster and to access cluster for new users. Implemented the Machine Learning algorithms using Python to predict the quantity a user might want to order for a specific item so we can automatically suggest using kinesis firehose and S3. Used AWS EMR to transform and move large amounts of data into and out of other AWS data stores and databases, such as S3 and DynamoDB. Import the data from different sources like HDFS/HBase into Spark RDD and perform computations using PySpark to generate the output response. Creating AWS Lambda functions with Boto3 to deregister unused AMIs in all application regions to reduce the cost for EC2 resources.

Designed and deployed end-to-end Azure data platforms leveraging services like Data Factory, Synapse Analytics, Data Lake Storage, Databricks and Cosmos DB.Migrated large on-prem SQL databases to Azure SQL Data Warehouse using performant ETL techniques. Implemented data security best practices.Built and managed complex Azure Data Factory pipelines with workflow orchestration, failure handling, automated restarts, data profiling and cleaning.Developed ETLs using Spark and Python to ingest and transform data from multiple sources into Azure data stores.Utilized Snowflake for building cloud Data Warehouses and analytics. Designed schemas, ETL logic and performed data modeling. Queried and optimized performance using Snow SQL.Gained hands-on experience using Terraform IaC for provisioning Azure resources and infrastructure as code.Performed data curation, cleansing and quality checks using Python and Spark data frames.Implemented data partitioning strategies for optimizing Azure Synapse and Data Lake Storage performance.Extract Transform and Load data from the different Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, Spark SQL and Python and Azure Data Lake Analytics.Worked on Production bugs especially involved in Azure Databricks Notebooks bugs and provided the new PySpark and Spark SQL logics to eliminate the bugs.Developed Notebooks and ETL Pipeline in Azure Data Factory (ADF) that process the data according to the job trigger.Hands-on experience on developing SQL Scripts for automation purposes.Strong understanding of Data Modeling in data warehouse environments such as star schema and snowflake schema.

Develop, design data models, data structures and ETL jobs for data acquisition and manipulation purposes. Expert in developing JSON Scripts for deploying the Pipeline in Azure Data Factory (ADF) that process the data. Expert in using Databricks with Azure Data Factory (ADF) to compute large volumes of data. Performed ETL operations in Azure Databricks by connecting to different relational database source systems using jdbc connectors. Developed Python scripts to do file validations in Databricks and automated the process using ADF. Developed an automated process in Azure cloud which can ingest data daily from web service and load in to Azure SQL DB. Developed Streaming pipelines using Azure Event Hubs and Stream Analytics to analyze data for dealer efficiency and open table counts for data coming in from IOT enabled poker and other pit tables. Analyzed data where it lives by Mounting Azure Data Lake and Blob to Databricks. Used Logic App to take decisional actions based on the workflow. Developed custom alerts using Azure Data Factory, SQLDB and Logic App. Developed Databricks ETL pipelines using notebooks, Spark Dataframes, Spark SQL and python scripting. Used Python and Shell scripts to Automate Teradata ELT and Admin activities. Worked on Tableau software for the reporting needs. Create pipelines in ADF using linked services to extract, transform and load data from multiple sources like Azure SQL, Blob storage and Azure SQL Data warehouse. Implement Azure Data Factory operations and deployment into Azure for moving data from on-premise into cloud. Develop batch processing solutions by using Data Factory and Azure Data bricks. Preparing ETL test strategy, designs and test plans to execute test cases for ETL and BI systems.

Developed Spark Applications by using Python and Implemented Apache Spark data processing Project to handle data from various RDBMS and Streaming sources. Expertise in ETL (Extract, Transform, Load) processes and data pipeline orchestration using Big Data Hadoop Technologies. Familiarity with Big Data technologies such as Hadoop, Spark, and Hive. Learner data model which gets the data from Kafka in real time and persist it to Cassandra. Developed Kafka consumer API in python for consuming data from Kafka topics. Consumed Extensible Markup Language (XML) messages using Kafka and processed the xml file using Spark Streaming to capture User Interface (UI) updates. Developed Preprocessing job using Spark Data frames to flatten JSON documents to flat file. Load D-Stream data into Spark RDD and do in memory data Computation to generate output response. Experienced in writing live Real-time Processing and core jobs using Spark Streaming with Kafka as a Data pipe-line system.

Optimizing of existing algorithms in Hadoop using Spark context, Spark-SQL, Data Frames and Pair RDD's. Developed Spark scripts by using Java, and Python shell commands as per the requirement. Involved with ingesting data received from various relational database providers, on HDFS for analysis and other big data operations. Experienced in performance tuning of Spark Applications for setting right Batch Interval time, correct level of Parallelism and memory tuning. Performed analysis on implementing Spark using Scala. Responsible for creating, modifying topics (Kafka Queues) as and when required with varying configurations involving replication factors and partitions. Used Hortonworks Apache Falcon for data management and pipeline process in the Hadoop cluster. Extracted files from Mongo DB through Sqoop and placed in HDFS and processed. Experience in Importing and exporting data into HDFS and Hive using Sqoop. Developed Flume Agents for loading and filtering the streaming data into HDFS. Experienced in handling data from different data sets, join them and pre-process using Pig join operations.Moving Bulk amount data into HBase using Map Reduce Integration. Developed Map Reduce programs to clean and aggregate the data. Developed HBase data model on top of HDFS data to perform real time analytics using Java API.Strong understanding of Hadoop eco system such as HDFS, Map Reduce, HBase, Zookeeper, Pig, Hadoop streaming, Sqoop, Oozie and Hive. Implement counters in HBase data to count total records on different tables.

Used Web Sphere for developing use cases, sequence diagrams and preliminary class diagrams for the system in UML. Extensively used Web Sphere Studio Application Developer for building, testing, and deploying applications. Developed the presentation layer and GUI framework in HTML, JSP and Client-Side validations were done. Involved in Java code, which generated XML document, which in turn used XSLT to translate the content into HTML to present to GUI. Implemented XQuery and XPath for querying and node selection based on the client input XML files to create Java Objects. Used Web Sphere to develop the Entity Beans where transaction persistence is required and JDBC was used to connect to the MySQL database.Developed the user interface using the JSP pages and DHTML to design the dynamic HTML pages. Developed Session Beans on Web Sphere for the transactions in the application.Utilized WSAD to create JSP, Servlets, and EJB that pulled information from a DB2 database and sent to a front-end GUI for end users. Used Struts framework for presentation layer. Implemented UI Layer with Struts, Struts validation, Struts Tiles, JSP, XML, CSS, HTML, JavaScript, AJAX and Angular JS. In the database major responsibilities includes creation of tables, triggers, stored procedures, sub-queries, joins, integrity constraints and views.Performed requirements analysis as per the client requirements. Proven ability to work with senior technical managers and staff to provide expert-level support for the installation, maintenance, upgrading, and administration of full-featured database management systems.