Distantmagic

Our mission is to provide open-source tools that enable the effective deployment and use of LLMs and ML models in production settings for developers and MLOps/LLMOps. Our key projects include: - Paddler, an open-source load balancer and reverse proxy optimized for servers running llama.cpp. Paddler ensures efficient request distribution with a stateful load balancer that monitors server slots and health, supporting dynamic scaling. - LLMOps Handbook (work in progress), a practical and advanced guide to LLMOps. It provides a solid understanding of large language models’ general concepts, deployment techniques, and software engineering practices. - Resonance, a PHP framework for building IO-intensive web applications and services featuring AI capabilities, a built-in web server, and integration with llama.cpp. It leverages asynchronous PHP and Swoole.