Lead Data Consultant
CurrentEnergy & Resources- Spearheaded the implementation of an ELT pipeline, leveraging cloud infrastructure to support a targeted flat file-oriented strategy for efficient data ingestion into reporting services. Utilized Striim for data extraction, while transformations were seamlessly executed by dbt and Snowflake using SQL. Azure Functions played a pivotal role in validating curated datasets, ensuring their availability as flat files for streamlined data consumption.- Executed infrastructure deployment with precision, adhering to CI/CD best practices using Terraform and Azure DevOps as key tools. Implemented a robust testing strategy with coverage thresholds in the release pipeline, ensuring the reliability and integrity of the deployed components.- Collaborated closely with diverse client stakeholders to establish comprehensive data requirements from the ground up, guaranteeing alignment with critical business logic and contributing to the success of the data-driven strategy.Finance and Insurance Sector- Implementation of a cutting-edge custom integration engine, designed to be system-agnostic and featuring advanced schema validation for structured and semi-structured (nested JSON files) data. Leveraged AWS CDK in Typescript with varying levels of abstraction to facilitate end-to-end infrastructure deployment, ensuring scalability and resource optimisation- Design of an event-oriented architecture using AWS EventBridge, incorporating automated schema inference through AWS Glue Crawlers and providing querying capabilities via AWS Athena. Enhanced end-user engagement through advanced notification streams using native AWS components (AWS SNS and Lambda), elevating system responsiveness and overall user experience- Engagement with CI/CD best practices through GitHub and complex, environment-specific Build and Release pipelines, fostering a culture of continuous improvement and rapid iteration. This resulted in quicker development cycles and agility.