Data Engineering Manager
CurrentAs the Data Engineering manager at Blackpool I am responsible for a small team of data engineers and together we ensure the critical flow of data into the Trust's data warehouse.The team is also responsible for the efficiency of the warehouse and perform regular checks on indexes, import speeds and troubleshoot each flow into the warehouse on failure.Powershell scripts are used to mitigate any issues with SQL Agents going offline at any point to ensure the integrity of the flows… Show more As the Data Engineering manager at Blackpool I am responsible for a small team of data engineers and together we ensure the critical flow of data into the Trust's data warehouse.The team is also responsible for the efficiency of the warehouse and perform regular checks on indexes, import speeds and troubleshoot each flow into the warehouse on failure.Powershell scripts are used to mitigate any issues with SQL Agents going offline at any point to ensure the integrity of the flows. Extensive logging is applied to the more impactful flows to monitor any bottlenecks and these are addressed when they arise.Although there is a separate Data Visualisation team responsible for reporting, the team also gets involved in building SSRS and Power-BI reports for use both within and outside the team and department.KPIs have been defined and captured for the monthly meetings which show data-flow health, failure counts, agent restart counts and much more. Indexing of the production tables and monitoring and reseeding those indexes is a basic requirement in the team. Recently work has been completed on a Master/Target server setup which has removed the requirement for jobs to setup across different servers, removing the risk of incorrect code stored on servers as there is now one single version of that job. Show less
