Junwei Li Email & Phone Number

Santa Clara, CA, US

• Modified vLLM source code to support a custom Mamba-like model architecture, integrating SSM and attention layers with sliding window and single-key head. Achieved an 8.3x increase in model throughput by enabling the.
• Applied quantization techniques like W8A8 and W4A16 to optimize LLMs, leading to a 78% improvement in inference speed by integrating them with vLLM and TensorRT-LLM. Published these models as services, handling up to 2.
• Enhanced Triton kernel for internal LLM development by optimizing the FP8 dequantization GEMM kernel using split-K, enabling improved performance across diverse GPU devices. Achieved a 23% increase in inference.

Atlanta, Georgia, United States

• (NeurIPS2024 Accepted) “AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deployment”
• Customized TensorRT-LLM to enable LLM structure pruning and conducted depth(layer num) and width(atten-head num) pruning for the llama2-7b model. Achieved a notable 37% enhancement in inference throughput and reduced.
• Developed AmoebaLLM, a framework enabling LLMs to instantly shrink into compressed subnets of varying structures using a mixture of LoRA, dynamic-programming-based depth shrinking, and in-place knowledge distillation.

China

2019/12 -- 2020/02 Winter Intern2020/06 -- 2020/08 Summer Intern

Master'S Degree, Computer Science, 3.9/4.0

Key Focus Areas: Efficient LLM Collaborating with Professor Celine Lin in LLM Efficiency improvement using Pruning, LoRA, Quantization, etc.

Bachelor'S Degree, Mechanical Engineering

U.S.News #168 in Global Ranking Selected Courses: Operating system, Data structure, Computer composition and system structure, Algorithm.