Jagdeep Singh Email and Phone Number

• Perform structural modifications using MapReduce, Hive and analyze data using visualization/reporting tools (Tableau).• Extending support to Oracle, Hadoop, & MongoDB along with DB2 to embrace multi DBMS support concept, and emb paradigm of management for Open Source Software, &Commodity Hardware to reduce costs.• Converting old oracle SQL/PL- SQL, Microsoft SQL server/ T- SQL written SQL to run on big data platforms using PYSPARK, SPARKSOL and HIVE with SAS and Python as programming platforms.• Was responsible for creating on-demand tables on S3 files using Lambda Functions and AWS Glue using Python.• Worked on the database migration from Netezza to AWS Redshift. Worked closely with the vendor on the plan and scripts needed to migrate the data.• Implemented e mappings using Normalizer and other transformations to process data from COBOL VSAM files by preparing respective Informatica Copybook for each and every file and then propagating data into Teradata warehouse.• Coordinated with IBM,Oracle, BEA, RedHat vendors to resolve issues related with middleware products.• Converting old oracle SQL/PL- SQL, Microsoft SQL server/ T- SQL written SQL to run on big data platforms using PYSPARK, SPARKSOL and HIVE with SAS and Python as programming platforms.

• Configured Oozie workflow to run multiple Hive and Pig jobs which run independently with time and data availability.• Developed a NIFI Workflow to pick up the data from SFTP server and send that to Kafka broker. • Used HUE for running Hive queries. Created partitions according to day using Hive to improve performance.• Used Docker to maintain Docker Hub, Images for middleware implementation and to automate the deployment process through Jenkin.• Used Databricks for encrypting data using server-side encryption.• Used Airflow to monitor and schedule the work• Converting SAS programs to python based for efficiency and utilizing the packages from NUMPY and PANDAS.• Involved in creating HDInsight cluster in Microsoft Azure Portal also created Events hub and Azure SQL Databases.• Configured Spark streaming to get ongoing information from the Kafka and store the stream information to HDFS.• Responsible for High Availability, & accessibility of databases. This effort involves 24X7 support of DB2 Sub systems to application interfaces on z/OS & Web• Perform Data Analysis on the analytical data present in AWS S3, AWS Redshift, Snowflake, Teradata using SQL, Python, Spark, Databricks.

• Build the Oozie pipeline which performs several actions like file move process, Sqoop the data from the source Teradata or SQL and exports into the hive staging tables and performing aggregations as per business requirements and loading into the main tables.• Running of Apache Hadoop, CDH and Map-R distros, dubbed Elastic Map Reduce(EMR) on (EC2).• Applied Apache Kafka to transform live streaming with the batch processing to generate reports.• Used AWS Data Pipeline to schedule an Amazon EMR cluster to clean and process web server logs stored in Amazon S3 bucket.• Experience in configuring the Zookeeper to coordinate the servers in clusters and to maintain the data consistency which is important for decision making in the process.• Worked on different data formats such as JSON, XML and performed machine learning algorithms in Python.• Application development using Hadoop Ecosystems such as Spark, Kafka, HDFS, HIVE, Oozie and Sqoop.• Involved in support for Amazon AWS and RDS to host static/media files and the database into Amazon Cloud.• Automated RabbitMQ cluster installations and configuration using Python/Bash.• Designed and Developed Real Time Data Ingestion frameworks to fetch data from Kafka to Hadoop.• Using partitioning and bucketing in HIVE to optimize queries.

• Used Hive to implement data warehouse and stored data into HDFS. Stored data into Hadoop clusters which are set up in AWS EMR. • Performed pig script which picks the data from one Hdfs path and performs aggregation and loads into another path which later pulls populates into another domain table. Converted this script into a jar and passed as parameter in Oozie script• Continuous monitoring and managing the Hadoop cluster through Cloudera Manager.• Performed Data Preparation by using Pig Latin to get the right data format needed. • Utilized the clinical data to generate features to describe the different illnesses by using LDA Topic Modelling.• Designed and developed weekly, monthly reports related to the Logistics and manufacturing departments using Teradata SQL.• Performed Data Visualization and Designed Dashboards with Tableau and generated complex reports including chars, summaries, and graphs to interpret the findings to the team and stakeholders. • Used Hive and created Hive tables and involved in data loading and writing Hive UDFs.• Used Sqoop to import data into HDFS and Hive from other data systems.• Used Git for version control with Data Engineer team and Data Scientists colleagues. Environment: Ubuntu, Hadoop, Spark, PySpark, Nifi, Talend, SparkSQL, SparkMLIib, Pig, Python, Tableau, GitHub, AWS EMR/EC2/S3, and Open CV.

• Gathered requirements from Business and documented for project development.• Prepared ETL standards, Naming conventions and wrote ETL flow documentation for Stage, ODS and Mart.• Prepared and maintained documentation for on-going projects.• Worked with Informatica Power Center for data processing and loading files.• Extensively worked with Informatica transformations.• Worked with SQL*Loader tool to load the bulk data into Database.• Developed mappings using Informatica to load data from sources such as Relational tables, Sequential files into the target system.• Designed and developed weekly, monthly reports related to the Logistics and manufacturing departments using Teradata SQL.• Created data maps in Informatica to extract data from Sequential files.• Coordinated design reviews, ETL code reviews with teammates.• Interacted with key users and assisted them with various data issues, understood data needs and assisted them with Data analysis.• Collect and link metadata from diverse sources, including relational databases and flat files.• Performed Unit, Integration and System testing of various jobs.• Extensively worked on UNIX Shell Scripting for file transfer and error logging.