Data Engineer
CurrentCurrently working to design and implement data engineering solutions for the consumer domain.Working to design solutions for batch and near real-time requirements to be used by Consent use cases, Decisions and Marketing Systems. This includes identifying requisite data points, designing and developing solutions to ingest data from various sources, cleansing data, transforming and loading data into data products ensuring data governance and data quality for downstream use cases.I develop batch and real-time data pipelines using Apache Beam, Python, and Google Cloud Platform components, including Cloud Functions, Dataflow, Pub/Sub and orchestration services like Composer, and workflows.I am proficient in applying DevOps principles in my work focussing on CI/CD using Gitlab and semantic release, using Terraform for Infrastructure as a code (IaC), automated testing, monitoring, logging and alerting. This includes building generic gitlab templates which could be reused to deploy code in various environments and various projects.I have experience in collecting metrics from various Google Cloud Platform services for monitoring purposes. Setting up dashboards to visually represent key metrics, system health and data staleness. Also, configure alerts and notifications to reduce manual efforts and improve mean time to recover.I am actively involved in GCP cost optimization projects, implementing strategies to reduce cloud expenses by 1 Million Pounds.