Leading Data Lake initiative for Supply Chain Systems used for Analytics purpose running on Hybrid Cloud.First in DSI to implement end to end Kafka connect architecture to production. Near real time data with 3~5 mins latencyKPIs can be onboarded and built quickly (within a sprint)↘︎ 91.6% decrease in Onboarding time↖︎ 15x faster speed of delivery with 70% development efforts automated100% data accuracy and completeness with eventual consistencyMTTD is immediate & MTTR is < 2 hours*Analysed and narrowed down the best architecture for the CDI initiative Reduced the development time for 20 tables from 1 quarter to 1 day (working to reduce to 2 hrs)Automated most of the development (70%) to increase the velocity of the team by 20%Partnered and executed Data Governance, Data Quality, Data Discovery, Data Lineage, Data Ingestion, Data TransformationTech Specialties: Big Data, Spark, Hadoop, Impala, Hive, Yarn, Cloudera, Presto, GCP, DataProc, GCE, Big Query, Airflow/Automic/Autosys, GCS, Vertica, Java, JEE, Web Service Technologies, Application DesignProcess Specialties: Agile, Scrum, JIRA, DevOps, CI-CD, Release Management, Incident Management, ITSM, Service NowManagement Specialties: Engineering Management, Project Planning and Execution, AOP, KTLO, QBR, Vendor Management, Stakeholder Management

• Strong knowledge on data profiling, data standardization, transformation, cleansing and best practices to apply on high volume data in establishing distributed, reliable and scalable data pipelines that seamlessly transfers data between HDFS and other storages like RDBMS/NOSQL Databases in real-time and batch.• performance improvement and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame, Dataset, Pair RDD's, Spark YARN for in-memory data processing like aggregation, filtering and grouping.• Strong Experience in working with Databases like DB2, SQL Server and MySQL and proficiency in writing complex SQL queries• Understanding about Kafka Cluster’s topics, producer, consumers, understanding about consume streaming data using spark streaming, from files and network logs.• Proven skills on benchmarking, performance optimizations, followed best practices in the development of spark jobs such as memory management, partitioning, broadcast variables and UI monitoring • Extensively worked on different serialization and file formats like CSV, XML, Avro, Parquet, Nested-Json, ORC, Text file, Sequence file using compression techniques Snappy and Gzip and store it in HDFS/Hive tables using Spark. • Unified Hive to analyse data ingested into HBase by using Hive-HBase integration and compute various metrics for reporting on the dashboard• Handled Hive performance optimization by using ORC/Parquet storage, Compression, Partitioning, Bucketing, Predicate Pushdown Data, CBO query analytics and compute analytics on top of tables, queries, and partitions• Experience in working with zookeeper, Oozie workflow and developed Oozie workflow to run scheduled batch cycles and present data for OLAP reports• Outline end-to-end strategy using Lambda and Kappa architecture approach. Proof-of-concept to determine feasibility and product evaluation of Big Data products.

Hadoop Data engineer with SQOOP, HIVE, SHELL,TABLEAU, AUTOSYS

Responsibilities on the projecta. Project monitoringb. Client interaction.c. Offshore Co-ordinationd. Team leade. Deliveryf. DesignActivities:• Lead team of 6 members• Interacted with client and co-ordinate offshore team. • Designed and developed COBOL/CICS programs to perform send files to external systems.• Created many subroutines which could be reused across the project.• Effective reviews and independent testing for correctness and quality standards, focusing on code optimization, defining standards and mentoring the team on de-bugging issues.

Responsibilities on the projecta. Project monitoringb. Codingc. Testingd. DesignActivities:• Designed and developed COBOL programs to perform data quality test, reconciliation of data, create aggregate reports for the external rating agencies (Fitch, SnP and Moodys). • Developed test scripts and the test data with the scrambled production like data.• Designed and developed COBOL programs to perform send files to external systems and created jobs using ICETOOL for data extraction and modification.• Created many subroutines which could be reused across the project.• Effective reviews and independent testing for correctness and quality standards, focusing on code optimization, defining standards and mentoring the team on de-bugging issues.