Azure Architect
Current● Designed and Developed ADF pipelines to orchestrate the pipelines flow leveraging Azure Data bricks.● Build scripts to Copy Data from onprem to Azure Cloud.● Designed Generic Metadata tables for auditing and monitoring purposes.● Developed Databricks notebooks for DDL and DML operations.● Experience in design and implementing a fully operational production grade large scale data solution on Snowflake Data warehouse● Build accelerators/ framework to perform unit testing effectively to match data and report issues.● Working on, Bigdata Components Hive Impala, Spark, Autosys and Azure.● Build ETL pipelines and processing using Bigdata, Pyspark to create feeds and load them in DataLake and into Snowflake.● Design and developed end-to-end ETL process from various source systems to Staging area, from staging to Data Marts and data load.● Designed ETL process using Talend Tool to load from Sources to Targets through data Transformations● Developed complex ETL jobs from various sources such as SQL server, Postgresql and other files and loaded into target databases using Talend ETL tool
