Senior Software Engineering - Tech Lead
CurrentKey Projects and Achievements- Algorithm Optimization: Refactored a clustering algorithm using PySpark, Scipy, and Numpy, achieving a 20x improvement in runtime and a 5x reduction in resource usage, resulting in annual cost savings of over $600K.- AI Pipeline Enhancement: Streamlined the core AI pipeline with PySpark, leading to a 15x improvement in runtime and resource utilization, saving over $250K annually.- Security Product Implementation: Led the development and deployment of a near real-time endpoint misbehavior detection system using AWS (Lambda, SQS, S3), Polars, Pandas, PySpark, GBT, and Airflow.- Streaming Framework Development: Played a key role as a core developer in creating a new streaming framework, focusing on I/O components and performance under scale load using Streaming PySpark, Hudi, Kafka, Airflow, and Locust.- Initiated and led several smaller projects, including: - Developing a framework for ML logging - Enhancing SQL queries, improvements related to SQLAlchemy usage and DB performance - Creating a feature to compare network performance across customers