Lead Data Engineer
CurrentCustomer: Cloud computing platform.Project Role: Lead Data Engineer; Cloud solution engineer.Projects: Corporate BI, Marketing analytics.Responsibilities:• Design and implementation of Declarative Data Warehouse framework. Implementation of declarative ETL approach helped significantly improve code quality and increase performance of Analytics team. Switch from Pentaho ETL to declarative ETL development helped analytics to fully focus on SQL logic instead of development in GUI.• Database performance tuning, performance and cost management. Declarative Data Warehouse framework collect performance and cost metrics for all operations performed during data loading and transformation. This metrics helped to detect most expensive queries and optimize them. Average monthly cost for GCP project reduced from 500+ thousands of dollars per month to 20-30 thousand per month.• Implementation of CI/CD practices. Declarative ETL approach helped to implement code review practices on a project and increased a speed of code delivery from development to production environments from week to couple days.• Designed ML model for prediction of technical support tickets counts. This model helped to optimize a workload of internal technical support team and to plan an optimal team headcount growth.• Designed and implemented solution for automatic scan of team’s repositories and network folders for any hardcoded passwords and password files.• Implemented a data clusterization for analysis of text-like information about expenses during employee’s business trips.• Prepared code for public opensource version of Declarative Data Warehouse framework.https://github.com/google/grizzly