• Improved the LLM training system tailored for Mixture-of-Experts atop the Fairseq framework, addressing GPU memory constraints and optimizing computational processes. Achieved an16x scalability improvement, reducing the required number of GPUs from 64 to 4 through parameter offloading to the CPU. Enhanced system throughput by up to 4.55x by integrating a calibrated algorithm for MoE, combined with pipelined CPU optimization and CUDA optimizations.• Built a distributed machine learning training system that accommodates large neural networks up to 3.9 billion parameters and achieves 100% GPU utilization with a unique pipelining technique. The system filled pipeline bubbles with the forward pass of the teacher network, resulting in a throughput increase of up to 12.15x when using 16 GPUs.• Improved inference speed of video neural representation, up to 2x (e.g. 40FPS) by decomposing the network into two networks and sharing the representations, enabling real-time decoding.• Accelerated training of an image self-supervised network by up to 1.4x was achieved through the use of a curriculum that progressively increases in difficulty with adjustments in learning rate, image size, and augmentation ratio.

Managed military assets data worth approximately $200k and 70 individuals’ identification information.