Teaching Assistant for CS 149: Parallel Computing

• Developed UI-JEPA, combining a JEPA video encoder with an LLM decoder to generate user intent from videos, outperforming SOTA MLLMs including Claude-3.5-sonnet and GPT-4 Turbo, while using just 5% of their model size, achieving 50.5x lower computational cost, and offering a 6.6x reduction in latency. https://arxiv.org/abs/2409.04081. • Created the first personalized planning benchmark, MultiCap, evaluating LLM ability to reason over personal context.• Developed the CAMPHOR collaborative agent framework to enhance LLM responses by actively seeking relevant information and leveraging function embeddings, achieving a 35% improvement in task completion F1 and reducing token usage by 98.3% through a prompt compression algorithm on MultiCap. https://arxiv.org/abs/2410.09407.

Teaching Assistant for CS 21SI: AI for Social Good

Under the guidance of Professor Diyi Yang:• Designed a taxonomy to represent social failures in human/agent-agent interactions in the physical world.• Created an evaluation framework using multi-dimensional criteria to assess an agent’s capability to respond appropriately to unexpected events.• Built an environment to simulate multi-turn action trajectories of agents in various real-world scenarios.• Established a benchmark to serve as a testbed and potential incubator for agents aware of physical social norms.

• Implemented a data compression algorithm using LLMs and LoRA tuning to compress sentences into embeddings of only ~25% size, and then recover information from compressed vectors through decoding, saving ~75% memory and time.• Implemented LoRA tuning and model parallelism for data compression algorithm using fairseq framework, allowing multi-GPU distributed training of LLaMa with flexible LoRA settings.• Explored speculative decoding techniques in causal language models, training a non-autoregressive drafter to decode multiple tokens and verify them simultaneously. Maintained high performance while boosting efficiency by 111%.

Under the guidance of Professor Sean (Xiang) Ren• Developed SwiftSage, which greatly enhances interactive task completion performance by employing dual-process theory and integrating the strengths of behavior cloning and prompting large language models (LLMs).• Improved code of ScienceWorld by fixing a time mismatch bug and extracting recent actions, seen objects and visited rooms from environment to observation, improving the score of baseline agent by 25%.• Experimented with SwiftSage, leading to an 87% performance boost and a 145% increase in cost-effectiveness.• Co-authored the paper SWIFTSAGE: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks accepted at NeurIPS. Details available at https://yuchenlin.xyz/swiftsage/.

Under the guidance of Professor Jie Tang• Implemented P-Tuning v2, which closes the gap for fine-tuning with only 1% tuned parameters across various settings by applying continuous prompts for every layer input of the pretrained Transformer.• Conducted extensive experiments on commonly used pre-trained models and NLU, showing consistent improvements for models ranging from 330M to 10B on hard sequence tasks.• Co-authored the paper P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks accepted at ACL. Code available at github.com/THUDM/P-tuning-v2.

• Conducted experiments to convert night-time images to day-time images to increase the accuracy of car detection using Generative Adversarial Network (GAN), in collaboration with the Ministry of Transport.• Improved code to directly fetch images using corresponding hyperlinks and fed into the GAN system, allowing future work to be done more smoothly and efficiently through the interface I wrote.