Data Engineering Technical Lead
CurrentResponsible for designing and implementing effective ETL solutions for structured and unstructured data using Data-bricks ( PySpark / Scala), Data Building Tool on AWS/Azure Infrastructure.Building a data catalog to store metadata, delta tables and enable data governance for Delta Lake.Utilized data analytics to identify trends and patterns in data and provide insights to stakeholders.Creation of data models and documented data flows to improve system architecture and data quality.Creation of Data-bricks Asset Bundles to deploy code to different instances and GitHub integration for version control.Optimizing data warehouse performance through indexing and query optimization