Senior Data Engineer
Current• Led enterprise level ETL architecture design initiatives & Developed Informatica mappings, sessions, and workflows to load transformed data into EDW from various source systems such as SQL Server, and Flat Files. • Extracted data from heterogeneous sources and performed complex business logic on data using Informatica transformations (e.g. Router, Normalizer, Lookup, Rank, Aggregator, Union, etc.) and\or SQL Scripts (e.g. BTEQ, and T SQL). • Designed Batch Audit Process in batch\shell script to monitor each ETL job along with reporting status which includes table name, start and finish time, number of rows loaded, status, etc and Developed a general-purpose Change Data Capture (CDC) process based on audit table for a standard incremental ETL process. • Collaborated with business intelligence and functional teams to gather information regarding the source and target mappings. • Data Profiling included the understanding of Data Lineage of Data flow from various feeds to various in stream tables and the columns. • Resolved high and critical defect fixes in the development of ETL processes generated by System Test & business rules change • Created new Mappings and modified existing Mapping /Workflows (Target - XML and DB), processed defect fixes, tested, validated data. Documented and implemented all the objects being used in ETL development into upper environments. • Developed scripts to load data to hive from HDFS and involved in ingesting data into Data Warehouse using various data loading techniques. • Developing data processing tasks using PySpark such as reading data from external sources, merge data, perform data enrichment and load in to target data destinations. • Designed, developed, and implemented complex SSIS packages, asynchronous ETL processing, Ad hoc reporting, and SSRS report server, and data mining in SSAS.