Senior Etl Engineer
CurrentJoined the team at the start of the modernization project (ETL/ELT). I progressed to a senior developer role and contributed to several other projects with a similar architecture.Responsibilities:- Data Extraction: Gather data from multiple sources (SFTP, S3, Kafka, IBM MQ)- Data Transformation: Preparing workflows for data transform, and map data into an internal global data format- Data Loading: Efficiently load processed data into target systems (RocksDB).Workflow Design & Automation: Develop ETL workflows and pipelines, automating data processes for efficiency and scalability (PowerShell)- Data Quality & Validation: Ensure the integrity, quality, and accuracy of data by performing arbitration, checks and validations throughout the ETL process.- Performance Optimization: Optimize ETL processes with large datasets for performance and memory efficiency.- Troubleshooting & Maintenance: Diagnose, debug, and fix issues in ETL pipelines, performing regular maintenance and updates.- Collaboration: Work with data analysts, architects, and business stakeholders to meet data requirements and ensure alignment with business goals (Scrum). Work with QA team in case of building automated unit and integration tests (Node.js)- Automation deploy: Collaboration with DevOps team in automation of solution deploy by updating CI/CD configAchievements:- built a repository of data transformations and workflows (~50 of solutions)- built a repository of scripts for data extraction (~30 of PowerShell scripts)- shared knowledge with the QA team to start making automated tests - updated integration tests to improve performance and flexibility- shared knowledge with the Support team, made tools for the Support team to ease issues debuging