Senior Data Engineer
Current• Functioning as Data Engineer in Deutsche Telekom’s digital products wing and helping the group and its 75+ subsidiaries undergo digital transformation and derive true data generated decisions.• Working directly on Deutsche Telekom’s market facing products like OneApp, OneShop, OneTV etc. and creating robust real-time data pipelines that help maintain the flow of data in the entire product ecosystem.• Involved in creating and optimizing the data pipelines and scheduling according to business requirements. Also involved in production support after application is live.• Involved in implementing Medallion architecture bronze, silver and gold layer for the Data warehouse and analytics use-case.• Worked on setting up spark streaming pipelines, scheduling, optimizing and keeping in mind fault tolerance and alerting.• Implemented streaming pipeline on Apache Nifi, configured alerts for them on Prometheus with high volumes of traffic approx. (200 – 300 million per day)• Optimized streaming process to reduce computation cost by using EMR efficiently via queuing and tags mechanism to approx. 50% and 30% for batch process.• Implemented DWH batch jobs using Glue and EMR, performed analysis using Athena, Redash.Tech Stack used and implemented: Apache Nifi, AWS glue, Redshift, Lambda, S3, EMR, Athena, Kafka, ELk , Redash, kubernetes, Docker, Schema Registry, Rabbit MQ, Jupyter, Rest API, KeyCloack, Tableau,Python,PySpark, Open search etc.
