Senior Data Engineer
London, Gb
Continuing to improve the quality and consumption of data at RVU, with a greater focus on cross-team initiatives, building infrastructure, and recruitment and training.Interviewed and recruited five new data engineers. For three of them, I was the primary contact during their onboarding and probation periods, which included responsibilities for managing HR requirements, creating stakeholder relationships, designing probation plans, and evaluating performance throughout.Designed and implemented novel tooling for monitoring and alerting data quality SLIs using BiqQuery metadata, Monte Carlo's graphQL API, Opsgenie, and Slack. This led to a dramatic improvement in data quality and trust - we are still gathering data for a proper evaluation, but current figures show an improvement in mean time to resolution of incidents of up to 80%. My colleagues and I presented the work on monitoring and alerting at Fivetran's Modern Data Stack Conference and Monte Carlo's Impact Data Observability Summit. The former is recorded here: https://resources.fivetran.com/mdsconference/mdscon-2021-how-to-build-a-single-customer-viewCreated detailed roadmaps for data engineering for multiple teams in the business, liaising with senior stakeholders throughout to ensure data engineering meets the needs of the business in all areas. Also responsible for quarterly reviews of objectives and roadmap with general managers.Implemented a new pricing model for our GCP estate resulting in an overall reduction of operational expenses of 50% annually while maintaining same or similar performance for 95% of query jobs.