How to Manage a GPU Fleet for LLM Inference
How to manage a GPU fleet for LLM inference: inventory, models, instances, access control and capacity planning across multiple servers.
Once one GPU server runs a model for your team, the next questions are operational. How do you manage a GPU fleet for LLM inference when there are five servers, three models and two hundred users? This article describes the work involved and how Loadcore handles each part.
The five jobs of fleet management
Running LLM inference across several servers comes down to five jobs:
- Knowing which servers exist and whether they are healthy.
- Getting the right model files onto the right servers.
- Starting, stopping and supervising model processes.
- Controlling who may use which model.
- Planning capacity as usage grows.
1. Inventory and health
Without tooling you track servers in a spreadsheet and find out about failures from users. Loadcore Agent runs on each Linux GPU server and sends a heartbeat every 30 seconds with its OS, architecture, GPU, RAM and public IP. It runs health checks on model instances every 15 seconds. The Console shows all servers and their state in one place.
To add a server, start the agent with a server key:
./loadcore-agent --api-key="lc_..."
On cloud GPU servers, the Console provides a startup script for user data or cloud-init. On first boot the server installs the agent as a system service, so a new machine joins your fleet without manual setup.
2. Distributing models
Model files are large, from roughly 18 GB for a quantized 27B to hundreds of GB for the biggest models. Copying them by hand does not scale. In the Console you search Hugging Face, choose a GGUF model and trigger a download on a specific server. The agent downloads directly from Hugging Face to ~/.loadcore/models and reports progress.
Size the model to the server:
| Model | Size | Fits |
|---|---|---|
| Gemma 4 31B | 18.3 GB (Q4_K_M) | One 24 GB or 48 GB card |
| gpt-oss 120B | 63.4 GB (MXFP4) | One 80 GB GPU |
| Qwen3 Coder Next | 48.5 GB (Q4) | One 80 GB GPU |
| GLM-5.3, DeepSeek V4.1 Flash | 200 to 500 GB | Multiple GPUs |
See the model library for the details of each.
3. Instances
An instance is a running llama.cpp process for one model, with its own port, context size and GPU layers. The agent starts, stops and supervises it. You can run several instances on one server if memory allows.
Splitting a model across servers
When a model is too large for one machine, a distributed instance splits it across up to 8 servers: one main server and up to 7 rpc servers, using llama.cpp RPC. Some limits to know:
- The servers must be on a local network.
- Only the main server needs the model file.
- It exists to fit bigger models, not to make them faster.
Adding throughput
For more throughput, do the opposite. Run the same model on several servers as separate instances and spread users across them. Use different departments or different tools pointing at different servers.
4. Access control
A shared fleet needs boundaries. Loadcore organizes people into an organization with admin and member roles, and into departments. Each department gets grants for specific models. Every member creates personal sk-lc- API keys and sees their own usage statistics. The agent validates keys against the control plane and caches the result for 5 minutes.
This means a legal team can use one model and an engineering team another, and you can revoke one person's key without touching anyone else. For regulated teams, see private AI for financial services and private AI for government.
5. Capacity planning
Watch three things: GPU memory per instance (weights plus KV cache), how many people use a model at once, and which departments grow fastest. When a model gets busy, add a server and start another instance of it.
Use the LLM cost calculator to compare fleet costs with hosted pricing, and read How Much Does It Cost to Self-Host an LLM for 200 Employees? for a worked example.
Capacity and availability
You stay in control of where models run. To add capacity, start the same model on another server. For availability, keep a second instance of your most important models on a different server, so your team can switch when one goes down for maintenance. Every server keeps serving on its own, because inference never depends on a central proxy.
The data path stays simple: inference traffic goes from tools directly to the agent on your server. Heartbeats, download progress, instance status and key checks go to the control plane. The control plane never proxies inference and never stores prompts or completions.
A simple operating routine
- Review the server list in the Console weekly for servers that stopped sending heartbeats.
- Keep one spare GPU server of the same size if the service matters to your business.
- Pin each department to a model, and review grants when people change roles.
- Test new models on one server before rolling them out.
Where to go next
Start with Getting Started with Self-Hosted LLMs on Your Own GPU Servers, see the on-premise LLM overview or open the docs.