Server-Side Software Engineer Internship
360 Wenku (Document Library) Data Migration and Filtration with LLMBeijing, China· Developed a unified document migration system for the 360 document library in collaboration with five document library companies. Leveraged Go language multi-threading, the system was deployed on Kubernetes, featuring no-code automatic migration and statistical tools. This approach enabled a successful migration of around 5 million new pieces of document data.· Coordinated with Microsoft, implementing 360 Smart Brain LLM to streamline content filtration of the document library. Provided Microsoft with high-quality and accurately-filtered Chinese data. Automated supply of 50,000 high-grade Chinese documents to Microsoft weekly.· Constructed an API interface to transform document content from the 360 document library into corpuses usable for model development. Enhanced data usability and quality for the training of the 360 large model.· Utilizing the company's proprietary XJSON data structure storage, integrated with SPDK, the initiative significantly raised corpus access speed - marking a 3.2 times improvement compared to using only the JSON format.