• Interacting with Business owners for requirement gatherings and analysis to define business & functional.• Implement data quality checks framework on top of production pipelines to ensure the accuracy and integrity of data stored in the data warehouse.• Developed and implemented an SCD Type 2 multi-layered data architecture for EHR systems, including stages for raw data ingestion, deduplication and normalization (prestage), staging (merging current and historical data), and unifying data from multiple sources.• led a proof of concept (PoC) project to integrate Iceberg tables into Snowflake, increasing analytical speed and data storage efficiency.• Improved Snowflake Query execution time by 30% by reworking Stored Procedures.• Fixed production Issues, worked on RCA (Root Cause Analysis) to make data accurate.

Worked with media client, to process there Big Datafile in spark over EMR. Writing the data to s3 for analysis using Athena. Loading the data-to-data lake for downstream team to work on the same.• Worked on Migration of Existing op-prem (hadoop) project to cloud ( spark / aws).• worked in the big data project which is using spark to do the Source data transformation and writing the data into S3 into parquet format and after that ingesting the data-to-Data Lake (Presto DB).• Worked on Next-Gen Orchestration Technology - AIRFLOW for scheduling and monitoring Jobs.• Worked on the Aws components like AWS lambda for creating the S3 Trigger Function that's doing the selective copy from Upstream to the downstream bucket used S3 bucket trigger Event and regex to do the selective copy.• Worked with diverse file formats including Parquet, JSON, TXT, CSV and ZIP, GZIP and GZ compression type• Experience in creating the Stored Procedure in PostgreSQL.

Worked as Big Data Hadoop developer. With responsibility of creating the Hive-SQL Scripts to run the data transformation process. Worked on the DQA framework to capture the data quality issue before sending the data to clients.• worked on multi-node Hadoop cluster, Involving the transformation of the data using Impala Queries and then writing the data into HDFS and then Storing the data from HDFS to Database using Scoop.• developed a stored procedure to transfer data from a temporary table to a final table in a partitioned format using sub-partitions, enabling the BI team to query data 50 times faster.• Worked on the Shell Scripting that copies the data from one NAS server to another using SFTP.• Experience in python based ETL jobs doing multiple transformations on the data frames.

Python ETL developer.