llama.cpp or vLLM: introducing Loadcore Edge and Loadcore Scale
Loadcore now runs models with llama.cpp and vLLM. Loadcore Edge is for individuals and small teams, Loadcore Scale for enterprises where many people share one model. Here is when to use which.
Loadcore started with one inference engine: llama.cpp. It runs almost anywhere, from a Mac on a desk to a rack of GPU servers, and it squeezes large models into little memory. For a person or a small team that is exactly right.
Larger organizations asked for something else. When 50 people use the same model at nine in the morning, the question is no longer whether the model fits, but whether everyone still gets a quick answer. That is the job vLLM was built for.
So Loadcore now runs both, and comes in two lines.
Loadcore Edge: llama.cpp for individuals and small teams
Loadcore Edge runs GGUF models with llama.cpp.
- Runs on almost anything. Linux, macOS and Windows, with NVIDIA, AMD or Apple Silicon GPUs.
- Fits big models in small memory. GGUF quantizations from 2-bit to 8-bit let a 27B model run on a single 24 GB card.
- Splits across servers. A distributed instance spreads one model over up to 8 servers on your local network with llama.cpp RPC.
- Quick to set up. The agent installs llama.cpp on first start.
Edge is the right choice when one person or a few people use a model, on a workstation, a Mac or a single GPU server.
Loadcore Scale: vLLM for enterprises
Loadcore Scale adds vLLM for Linux servers with NVIDIA or AMD data center GPUs.
- Many people, one model. vLLM batches requests from many users together continuously, so response times stay steady when the whole company is online.
- All GPUs in a server. One model can spread over every GPU in a server with tensor parallelism, for larger models and more throughput.
- Full-precision and data center formats. vLLM loads Hugging Face model repositories in BF16, FP8, AWQ and GPTQ.
- llama.cpp included. Smaller models and GGUF files keep running with llama.cpp on the same servers.
Scale is the right choice when a model is shared by a department or the whole organization.
What stays the same
Both lines use the same console, the same departments and per-model access rules, the same personal sk-lc- keys and the same chat. Every instance serves the same OpenAI-compatible endpoint on the agent, so opencode, aider, Continue and your own applications work unchanged. Moving a model from llama.cpp to vLLM changes nothing for the people who use it.
Prompts and completions still never pass through Loadcore. The control plane only learns which instances run where.
Choosing the engine
You choose the engine per instance, not per server. When you start an instance, Loadcore suggests one:
| Situation | Suggested engine |
|---|---|
| Several people have access to the model, the server supports vLLM and the model is a Hugging Face repository | vLLM |
| One or a few users | llama.cpp |
| A Mac, Windows machine or consumer GPU | llama.cpp |
| A GGUF file, for example a 2-bit or 4-bit quantization | llama.cpp |
| A model too large for one server | llama.cpp with a distributed instance |
You can always override the suggestion.
For vLLM instances you set the context size, how many GPUs the model uses, the share of GPU memory vLLM may take (0.90 by default) and how many requests it handles at once. For llama.cpp instances you set the context size and GPU layers, as before.
How vLLM gets onto your server
There is nothing to install by hand. The first time you start a vLLM instance, the agent sets up vLLM and shows the progress in the console, like a model download:
- With Docker and the NVIDIA Container Toolkit installed, it uses the official vLLM container image.
- Without Docker, it creates a Python environment for vLLM that the agent manages itself.
The agent keeps each model process on the loopback interface behind its own key, exactly as it does for llama.cpp, and checks every request against your access rules.
Which line do I need?
- Starter includes Loadcore Edge.
- Professional and Enterprise include Loadcore Scale, with both engines.
Not sure which fits your team? The model guides now list vLLM settings and GPU fits next to the GGUF sizes, for example for Qwen3.8 27B and gpt-oss 120B. The docs describe the requirements for both engines. Questions about your setup are welcome at sales@loadcore.io.