 Worked on architecture and designed the FHIR project which included AWS services, AWS Glue, Databricks and Elastic Search Worked on Cloud Centre of Excellence team for Project design use cases and POC. Played the cloud architect role in migration application to AWS cloud

 Developed AWS Glue jobs using pyspark and scheduled using Stepfunctions, Airflow. Created AWS Systems Manager for ec2 instance deployment and orchestration. Worked on AWS Databricks for ETL transformations using Delta tables, Delta live tables. Used Glue Databrewer for data cleaning, normalize and prepare data.  Created Tableau reports from hive tables for showcase to clients. Worked on performance improvement on heavy load ETL data loading operations. Used Terraform for creation of AWS resources using Jenkins pipeline. Creation of IAM policies and IAM roles using Terraform.

 Developed Java spring boot application for processing real time transaction data in kafka and deployed in inhouse cloud environment. Created kafka topics, producer and consumer groups and worked on sending and retrieving real time data. Worked on kafka offset handling, metadata header and retrying mechanism of messages upon failure using dead letter queue. Created frameworks using Spark for flattening and mapping nested avro messages. Used mapping diagnostic context(MDC) from application logging. Used Cassandra as a service to load the transaction data into Cassandra. Responsible for building scalable distributed data solutions using Hadoop. Worked on ingestion of kafka streaming messages into Hadoop environment. Developed transformations using spark java for loading data from conformed to semantic zones. Involved in creating Hive tables, and loading and analyzing data using hive queries Configured continuous delivery pipelines for deploying micro services using CI/CD Jenkins. Developed Hive queries to process the data and implemented Partitioning, Dynamic Partitions, Buckets in HIVE. Classifying the data for protection group and ranger policies. Performed functional testing using Junit and written cucumber test cases. Created Tableau reports from hive tables for showcase to clients.

 Used Spark-Streaming APIs to perform necessary transformations and actions for credit reporting model which gets the data from Kafka in near real time. Consumed avro/json formats of data and parsed it and loaded to hive tables in orc format for data consumption. Implemented the Hadoop streaming data ingesting using spark java. Developed transformations using Spark java and Scala and loaded data in semantic zone. Involved in high level design of processing SOR files from edge node to conformed and semantic zone. Used Spark API over Cloudera Hadoop YARN to perform analytics on data in Hive. Created Managed, External and ORC HIVE tables on top of Hadoop file system. History migration from Teradata to Hadoop using Sqoop. Implemented schema extraction for Avro file Formats. Experienced in handling large datasets using Partitions, Spark in Memory capabilities, Effective & efficient Joins, Transformations and other during ingestion process itself. Created Unix automated scripts for Hive and spark. Created restful APIs using Java spring boot for producing messages. Scheduling reports using ControlM job schedule.

Acts as team lead for informatica/Teradata project in onshore. Actively involved in Project Estimates, Requirement gathering and Design, code development with informatica and teradata knowledge. Configuring on Informatica data quality for data cleansing, data matching, data conversion and exception handling. Creation of stored procedures and calling them from informatica power centre for processing in Teradata. Worked in creation of Informatica mappings and BTeq Scripts for semantic solutions. Worked on Performance Tuning to optimize the Session performance by utilizing, Partitioning, Pushdown optimization, pre, and post stored procedures to drop and build constraints. Worked on Audit login, Error login and Reference check while loading data to Data warehouse. Creation of Tivoli jobs and performs unit and Integration testing. Created Unix scripts for file processing. Performed Unit and Integration testing. Also supported users during UAT testing.  Build the Dimension & Facts tables load process and reporting process using Informatica. Completely Involved in development of informatica mappings for stage 1 and stage 2 process based on study of Abinitio graphs. Using IDQ for Data cleansing in staging layers. Analysed the skewness of tables and implementing possible solutions after discussing with cards SMEs Analyzing the PDO queries and providing best solutions to increase the performance of mappings. Working on stored procedures for processing large result sets. Worked in optimization of mappings. Involved in development of SOR layer for load ready files. Running Control M jobs and performs unit and Integration testing.

 Build the Dimension & Facts tables load process and reporting process using Consolidated sales ordered data using Informatica and loaded to aggregate tables for reporting.  Worked with the DBA to improve ETL performance and query performance by collecting statistics and defining relevant indexes on database tables Developed various complex Mapplets and Stored procedure to facilitate loading of data on a weekly and monthly basis. Worked on Various transformations such as Joiner, Filter, Update Strategy, Expression, Router, Lookup, Union, and Aggregator. Administered and developed monitoring scripts for power center server process and workflow jobs. Designed and developed enhanced data models and ETL applications to meet the business requirement. Used Informatica Power Connect to extract data from third party databases, web services, and XML. Mapping type 1 & type 2 Slowly Changing Dimension (SCD) using Informatica. Timely delivery of ETL tasks is performed using workflow manager and scheduler. Worked closely with DBA and developers during planning, analyzing, developing and testing phase of the project.  Involved in the Performance Tuning of Database and Informatica. Improved performance by identifying and rectifying the performance bottlenecks.

 Development of eVehicle website using javascript and J2EE and Application Deployment. Extensively worked in the performance tuning of programs, ETL procedures, and processes. Configured Informatica Power Exchange connection and navigator.  Created Registration, Data Map, configured Real-Time mapping and workflows for real-time data processing using CDC option of Informatica Power Exchange.