Pavan Ch Email & Phone Number

Mclean, Va, Us

Project Description: Federal Deposit Insurance Corporation (FDIC) has created a new record keeping requirement (Part 370) for the largest insured depository institutions (IDI) - compliance is required by April 1, 2020. This rule requires large IDIs to now determine & calculate insured/uninsured balances and to uniquely identify fiduciary owners and beneficiaries via reporting within 24 hours. The ultimate goal is full compliance, the key focus for a large domestic bank is to properly prioritize remediation efforts regarding data and system gaps based on their impact to make a deposit insurance calculation.Responsibilities:• Loaded D-Stream data into Spark RDD from S3(Simple Storage Service – AWS Service) and do in memory data Computation to generate Output response based on the transformations and actions.• Worked and learned a great deal from AWS Cloud services like EMR, Step Functions, EC2, S3, RDS, Cloud Watch, Lambda, Athena, GLUE Setting up IAM Roles.• Worked on a great deal using ELASTIC MAP REDUCE and setup Hadoop environment on AWS EC2 Instances also responsible for EMR cluster trouble shooting.• Worked on PySpark by modifying the existing Python scripts that reads the input data, apply Spark transformations and write the result to a file using python.• Used the Presto Query engine to run the ad-hoc queries using the created views, by on boarding tables meta data to AWS Glue. Used Red Dash UI in order to execute the queries on Datasets stored on Amazon S3.• Set up the Spark testing base using the Holden-Karau Spark Testing Base from scratch for testing the data transformation logics on QA before promoting the code to production.

Us

Built and deployed the scalable distributed Hadoop cluster running on Hortonworks Data Platform (HDP 2.6) in the custom stack on the physical Data Center which is on-premise and later migrated it to AWS. • Developed Kafka consumer's API in Scala for consuming data from Kafka topics and Consumed XML messages using Kafka and processed xml file using Spark Streaming to capture UI updates.• Worked and learned a great deal from AWS Cloud services like EC2, S3, EBS, RDS and VPC.• Migrated an existing on-premises, Physical-DC application to AWS. Used AWS services like EC2 and S3 for small data sets processing and storage, Experienced in Maintaining the Hadoop cluster on AWS EMR. • Imported data from AWS S3 to Spark RDD, performed transformations and actions on RDD's using Scala.• Worked with ELASTIC MAPREDUCE and setup Hadoop environment on AWS EC2 Instances.Involved in creating Data Lake by extracting Application, customer’s data from various data sources to HDFS/HBase which include data from Relational Databases and logging data from servers and Micro services.