Junwei Li Email & Phone Number

Santa Clara, Ca, Us

• Modified vLLM source code to support a custom Mamba-like model architecture, integrating SSM and attention layers with sliding window and single-key head. Achieved an 8.3x increase in model throughput by enabling the new structure to run efficiently on vLLM.• Applied quantization techniques like W8A8 and W4A16 to optimize LLMs, leading to a 78% improvement in inference speed by integrating them with vLLM and TensorRT-LLM. Published these models as services, handling up to 2 million requests per day.• Enhanced Triton kernel for internal LLM development by optimizing the FP8 dequantization GEMM kernel using split-K, enabling improved performance across diverse GPU devices. Achieved a 23% increase in inference throughput

Atlanta, Georgia, United States

• (NeurIPS2024 Accepted) “AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deployment”• Customized TensorRT-LLM to enable LLM structure pruning and conducted depth(layer num) and width(atten-head num) pruning for the llama2-7b model. Achieved a notable 37% enhancement in inference throughput and reduced model size by 35%.• Developed AmoebaLLM, a framework enabling LLMs to instantly shrink into compressed subnets of varying structures using a mixture of LoRA, dynamic-programming-based depth shrinking, and in-place knowledge distillation.

China

2019/12 -- 2020/02 Winter Intern2020/06 -- 2020/08 Summer Intern

Master'S Degree, Computer Science, 3.9/4.0

Key Focus Areas: Efficient LLM Collaborating with Professor Celine Lin in LLM Efficiency improvement using Pruning, LoRA, Quantization, etc.

Bachelor'S Degree, Mechanical Engineering

U.S.News #168 in Global Ranking Selected Courses: Operating system, Data structure, Computer composition and system structure, Algorithm.