 Creating Database Objects on Snowflake including tables, views, and schemas, to support business requirements. Prepared Source mapping document by interacting with source teams and stakeholders to process the required information from the source input data. Designing and developing Data Pipelines for Data Ingestion and Transformation using PySpark/Spark SQL Developed data pipelines using Databricks to load data from multiple sources to Snowflake. Performance Tuning- Monitor and optimize query performance in Snowflake to ensure efficient data retrieval. Created jobs for production deployment and monitor ETL jobs to ensure they run successfully and meet SLAs during postproduction support.  Documentation of the process and jobs and creating confluence pages for reference.  Providing support in resolving post-migration issues. CI/CD pipeline using GitLab, ensuring smooth and automated deployment of ETL jobs.

 Creation of primary objects (Tables, Views, etc) required for the application on Snowflake. Development of Spark code using PySpark and Spark-SQL. Loading source data into S3 raw zone using Pyspark jobs. Transformed and loaded cleansed data into Snowflake tables. Carrying out Performance Tuning of Jobs, Stages, Sources and Targets. Support Unit Testing and working with QA / UAT Team to resolve defects. Writing / reviewing of Detail Design Document with Mapping Specifications & Unit & Integration Test Plans / Test Cases. Creating & scheduling Data pipelines jobs through AutoSys, guaranteeing smooth and timely ETL operations.

 Creating the DB Objects, deciding Primary / Secondary index, creating Join index, Partitioned Primary Index, Compare Load Utilities to determine the best load scenario. Writing scripts to importing the data from source files like flat file using the Teradata load utilities. Worked on data Extraction, Transformation and Loading from source to target system using BTEQ. Expertise is selecting Primary Index (UPI, NUPI) for a table for uniform data distribution across the AMP’s and Compressing Columns for efficient utilization of available disk space. Very strong on Teradata SQL query optimization and tuning techniques like: EXPLAIN feature, COLLECT STATISTICS option, Secondary Indexes (USI, NUSI), Partition Primary Index (PPI), Volatile and global temporary tables, selecting Primary Index (UPI, NUPI) for a table for uniform data distribution across the AMP’s and Compressing Columns for efficient utilization of available disk space. Extensively used Teradata Load and Unload utilities such as OLE Load, Multi Load, Fast Export and Bulk Load stages in Jobs for loading/extracting huge data volumes. Weekly meetings with technical collaborators and active participation in code review sessions with Team.

 Developing Ingesting pipelines to import data into HDFS using Sqoop from Teradata database. Built jobs to process the Data on HDFS and load into Hive Tables using Hive SQLs. Created views on Lake Tables for Tableau reporting team to create reports/dashboards. Loading data into HBase for Mart specific users. Worked on SVN for Version control, and Oozie for workflow management. Performance tuning of HQL jobs using various Optimization techniques Responsible for overseeing the Quality procedures related to the project.

 Worked on Huge file data sets ingestion using Talend.  Developed jobs for processing the Data and load into Teradata Tables Worked on Github, Autosys for deployments, orchestration, and scheduling of jobs. Perform rigorous data analysis to identify any inconsistencies or data quality issues and report with Architects and leads.