 Developed Python Spark streaming scripts to load raw files and corresponding.  Used snowpark for extracting the data from source and loading it to Enterprise snowflake. Implemented PySpark logic to transform and process various formats of data like XLS, XLS, JSON, and TXT. Elaborated Python Scripts to fetch/get S3 files using Boto3 module.  Built scripts to load PySpark processed files into Redshift DB and used diverse PySpark logics.  Developed MapReduce programs to cleanse the data in HDFS obtained from heterogeneous data sources. Processed metadata files into AWS S3 and Elastic search cluster. Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in Azure Databricks.  Used Azure Data Factory, to create and schedule data-driven workflows (called pipelines) that can ingest data from disparate data stores Created Hive Generic UDF's to process business logic that varies based on policy.  Moved Relational Data base data using Sqoop into Hive Dynamic partition tables using staging tables. Monitoring Cluster using Cloudera manager. Included migration of existing applications and development of new applications using AWS cloud services. Developed Python Scripts to get the recent S3 keys from Elastic search.

 Involved in complete Implementation lifecycle, specialized in writing custom MapReduce, Pig and Hive.  Have been using NiFi for transferring data from source to destination and Responsible for handling batch as well as Real-time Spark jobs through NiFi. Developed micro-services using Python scripts in Spark Data Frame API’s for the semantic layer. Developed Spark scripts by using Scala as per the requirement. Responsible to manage data coming from different sources and involved in HDFS maintenance and loading of structured and unstructured data.  Developed a data pipeline using Kafka, Spark and Hive to ingest, transform and analyzing data. Used Scala to convert Hive/SQL queries into RDD transformations in Apache Spark.  Built the complete data ingestion pipeline using NiFi which POST’s flow file through invoke HTTP processor to our Micro services hosted inside the Docker containers. Involved in creating Hive tables, loading with data and writing Hive queries which will run internally in MapReduce way. Designed and implemented MapReduce based large-scale parallel relation-learning system. Processed Multiple Data sources input to same Reducer using Generic Writable and Multi Input format. Performed data profiling and transformation on the raw data using Pig and Python. Visualize the HDFS data to customer using BI tool with the help of Hive ODBC Driver.  Created Hive Generic UDF's to process business logic that varies based on policy.  Moved Relational Data base data using Sqoop into Hive Dynamic partition tables using staging tables. Monitoring Cluster using Cloudera manager. Develop predictive analytic using Apache Spark Scala APIs.  Implemented MapReduce counters to gather metrics of good records and bad records. Built data governance processes, procedures, and control for Data Platform using Nifi.

 Worked on python API to extract data from RDBMS and create dataframes to save as table and submit in spark jobs. Worked on Sqoop import and export from Azure, SQL Server, Oracle, DB2 and Teradata and configure to run in parallel in shell scripting. Extensive working knowledge in Core java concepts, Collections and Web based, Client/Server Applications. Loaded large sets of structured, semi structured data from Swift Object Store to HDFS in Edge node with DISTCP command and created staging table. Worked on POWER BI to create reporting graphs and cards for analysis for different years and months based on different KPI’s requirement. Worked on TDCH connector to export table to Teradata and configuring the job in shell scripting. Experience to create HQL scripts and python scripts to submit in pyspark jobs with optimize resources. Created shell scripts to perform file transfer between Windows server and file server. Extensively worked on error handling in shell scripting as well as in python depending on downstream jobs requirement. Extensively worked on Shell Scripting to configure the spark jobs and Sqoop jobs from databases. Experience of handling huge datasets in Spark Jobs and submit parallelly to run as small subsets of jobs with the help of partitioning. Performed transformations between many hive tables to create final feeds. Exploring with the Spark improving the performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame, and Pair RDD's. Written SQL queries for data analysis to meet the business requirements and perform validations on it. Hands on experience on troubleshooting spark errors related to executors and java heap memory. Worked on converting the data sets to Hive ORC format and partitioning the data on week number basis.

 Involved in Requirement Gathering to connect with BA. Working Closely with BA & Client for creating technical Documents like High-Level Design and low-Level Design specifications. Experienced on loading and transforming of large sets of structured data, semis structured data and unstructured data. Imported data using Sqoop to load data from MySQL to HDFS on regular basis. Developing RDDS to schedule various Hadoop Program. Written SPARK SQL Queries for data analysis to meet the business requirements. Experienced in defining job flows. Cluster coordination services through Kafka and Zookeeper. Serializing JSON data and storing the data into tables using Spark SQL. Writing Shell scripts to automate the process flow. Storing the extracted data into HDFS using Flume Experienced in multiple file formats including XML, JSON, CSV and other compressed file formats Experienced writing queries in Spark SQL using Scala Communicated all issues and participated in weekly strategy meetings. Collaborated with the infrastructure, network, database, application, to ensure data quality and availability. Experience in Daily production support to monitor and trouble shoots Hadoop/Hive jobs Support/Troubleshoot hive programs running on the cluster and Involved in fixing issues arising out of duration testing. Prepare daily and weekly project status report and share it with the client. Implemented Data Integrity and Data Quality checks in Hadoop using Hive and Linux scripts

 Involved in Requirement Gathering to connect with BA. Working Closely with BA & Client for creating technical Documents like High-Level Design and low-Level Design specifications. Experienced on loading and transforming of large sets of structured data, semis structured data and unstructured data. Imported data using Sqoop to load data from MySQL to HDFS on regular basis. Developing RDDS to schedule various Hadoop Program. Written SPARK SQL Queries for data analysis to meet the business requirements. Experienced in defining job flows. Cluster coordination services through Kafka and Zookeeper. Serializing JSON data and storing the data into tables using Spark SQL. Writing Shell scripts to automate the process flow. Storing the extracted data into HDFS using Flume Experienced in multiple file formats including XML, JSON, CSV and other compressed file formats Experienced writing queries in Spark SQL using Scala Communicated all issues and participated in weekly strategy meetings. Collaborated with the infrastructure, network, database, application, to ensure data quality and availability. Experience in Daily production support to monitor and trouble shoots Hadoop/Hive jobs Support/Troubleshoot hive programs running on the cluster and Involved in fixing issues arising out of duration testing.

 Managing over 20 critical applications single. Configuring Dataguard for OLTP databases. Upgrading databases to 12C. Work with Application team to performance related issues. Rebuilding of Indexes for better performance, maintenance of Oracle Database. Generated performance reports and Daily health checkup of the database using utilities like AWR, Statspack to gather performance statistics. Identified and tuned poor SQL statements using EXPLAIN PLAN, SQL TRACE and TKPROF, analyzed tables, indexes for improving the performance of the Query. Troubleshooting various issues like database connectivity to users, privileges issue. Created users and allocated appropriate table space quotas with necessary privileges and roles for all databases. Wrote script to monitor the database with shell and PL/SQL code or SQL code such as procedure, function and package.  Created or cloned the oracle Instance and databases on ASM. Performed database cloning and re-location activities. Managed tablespaces, data files, redo logs, tables and its segments. Maintained data integrity also managed profiles, resources and password security manage Users, privileges and roles.