Data Engineer
CurrentThis was a varied role withing the risk and analytics team combing the following tasks:Review the team’s core SQL Server databases for configuration and performance issues, documenting and making the appropriate fixes to improve reliability and scalability. Catalogue and document data assets in Microsoft Purview, from various sources including Microsoft Fabric, ADF and SQL Server on Azure VM. Design and implement the team’s first Microsoft Fabric workspace. This involved researching all available functionality, to understand any limitations and to define the most appropriate design focusing on reliability, ease of use, scalability and running costs. The data sets added to Fabric came from a variety of sources including blob storage and SQL Server on Azure VM. The data processing methods trialled included ADF pipelines, Fabric pipelines and Fabric notebooks using python and Spark SQL. The various methods of adding Fabric items into source control were also trialled along with the use of Fabric’s deployment pipelines.