Data Engineering Tech Lead
Current- Acted as Tech Lead and owned the Customer Data Platform (CDP) project end-to-end, managing multiple stakeholders and ensuring successful delivery of key objectives. The CDP enabled deep insights into user behavior and accelerated efforts for the marketing and data science teams.- Contributed to the development of Karavan, an internal Spark application framework designed to enhance pipeline quality and reduce development time.- Led the development of user behavior tracking pipelines and contributed to building a robust data platform to support large-scale data processing.- Developed and optimized pipeline management tools for orchestration and monitoring, improving the operational efficiency of data engineering workflows.- Revised and re-architected the data platform, leading the team in implementing a more scalable and efficient system, improving overall performance and maintainability.- Maintained and optimized key infrastructure components, including Spark clusters, Kafka clusters, HDFS, Pinot, and ClickHouse, ensuring platform stability and scalability.- Collaborated with data scientists on projects such as search and recommendation systems, image and text-based projects, managing production model deployments, and optimizing data science pipelines for real-time and batch processing.
