Paddler
Open SourceApache-2.0LLM load balancer and serving platform for self-hosting LLMs at scale 🏓🦙 Alternative to projects like llm
Description
Open-source LLM load balancer and serving platform for self-hosting LLMs at scale 🏓🦙 Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU.
Preview
Platforms
LinuxmacOSDocker
Replaces the following proprietary tools
ChatGPT API→Paddler
Midjourney→Paddler
DALL-E→Paddler
Self-Hosting
Paddler can be self-hosted on your own infrastructure. Visit the official documentation for installation instructions.
Documentation →Categories
Tags
#ai#llamacpp#llm#llmops#load-balancer