Part of the Insights & Data team that managed the client’s ETL pipelines and EDW resources on Azure Databricks using Python and SQL and in Synapse Analytics using SQL, DataFactory and PowerBI. Monitored Azure Databricks pipelines daily for execution and performance. Provided support to the L1 team for any execution downtimes, errors or pipeline failures. Worked on the performance tuning of Azure Databricks pipelines such as filtering out-of-range data(flag checking for greater than > 5 years data),removal of duplicates and to add features like converting generic time zones to Australia/ New Zealand time zone and date-timestamp format on user request.Worked on the existing pipelines to add features like string formatting, correction of misaligned data, truncate and trimming of outliers, white spaces and special characters.  Created Hive Metastore tables for backup and maintenance purposes.Worked on the Azure Synapse part of the data pipeline. Used SQL on the Dedicated SQL Pool table data to ensure consistency and concurrency. Eg: data was matched with another database table check “change-of-address” and update when required.