Loadcore
All posts
Nov 6, 2026 · Loadcore Team

Getting Started with Self-Hosted LLMs on Your Own GPU Servers

A practical guide to self-hosted LLMs: pick a model, size your GPU server, run inference on your own hardware and give your team an OpenAI-compatible endpoint.

self-hosted LLMgetting startedGPU serversguide

Self-hosted LLMs let a company run open-weight language models on GPU servers it controls, instead of sending every prompt to a hosted provider. This guide walks through what you need to get from nothing to a working, team-wide endpoint: a model, a GPU server, an inference runtime and a way to manage access.

If you are still deciding whether self-hosting makes sense, read Can Open-Source AI Replace ChatGPT in Your Company? first. If you already know you want it, continue here.

What self-hosting actually means

You download model weights (the files that make up the model), load them into GPU memory on a server you control, and expose an API that your tools can call. Prompts and completions never leave that server.

The pieces are:

  • A model. Open-weight models such as gpt-oss, Qwen, Gemma or DeepSeek, usually in GGUF format.
  • A GPU server. Your own hardware, a colocation machine or a rented cloud GPU.
  • An inference runtime. llama.cpp is the one Loadcore builds on.
  • Access control and visibility. Who may use which model, with which key.

For a broader overview of the on-premise approach, see on-premise LLM.

Step 1: Pick a model that fits your hardware

The first constraint is GPU memory. The model file must fit in VRAM, plus extra room for the KV cache, which grows with context length and the number of parallel requests.

ModelSize on diskHardware fit
Gemma 4 31B18.3 GB (Q4_K_M) to 33.2 GB (Q8_0)One 24 GB card at Q4_K_M, one 48 GB card at Q8_0
Qwen3.8 27B17.6 GB at Q4One 24 GB class card
gpt-oss 120B63.4 GB (MXFP4)One 80 GB GPU
Qwen3 Coder Next48.5 GB at Q4One 80 GB GPU
GLM-5.3, DeepSeek V4.1 Flash200 to 500 GBMultiple GPUs

Sizes come from our model library. Quantization (Q4, Q8 and so on) trades a little quality for a much smaller file. For most business use, Q4 to Q6 is a sensible starting point.

A practical rule: start with a mid-sized model such as Gemma 4 31B on a single 48 GB card. It is easy to operate and handles drafting, summarizing, translation and Q&A well. Move up to a larger model only when your own testing shows you need it.

Step 2: Get a GPU server

You have three realistic options:

  1. Hardware you own. Best when you already have GPU servers or a data center and want full control.
  2. Colocation. Your hardware in someone else's facility, useful for choosing a specific country.
  3. Rented cloud GPUs. The fastest way to start. You choose the provider and region, for example an EU region if data location matters to you. See the GDPR guide.

Loadcore's agent runs on Linux GPU servers, so any of these works.

Step 3: Install the agent and start a model

Loadcore Agent is a single Go binary with no runtime dependencies. On first run it installs llama.cpp for you. You start it with a server key from the Console:

./loadcore-agent --api-key="lc_..."

For cloud servers, the Console generates a startup script you paste into the user data or cloud-init field. On first boot the server installs the agent as a system service and registers itself.

Then, in the Console at console.loadcore.io:

  1. Open Models and search Hugging Face for a GGUF model.
  2. Start the download. The agent fetches it directly to ~/.loadcore/models and reports progress.
  3. Start an instance, choosing the port, context size and number of GPU layers.

The agent supervises the llama.cpp process, runs a health check every 15 seconds and sends a heartbeat every 30 seconds with OS, architecture, GPU, RAM and public IP.

Step 4: Connect your tools

The agent serves an OpenAI-compatible API on port 9090 by default: GET /v1/models, POST /v1/chat/completions and POST /v1/completions. Point any compatible client at it:

curl http://your-server:9090/v1/chat/completions \
  -H "Authorization: Bearer sk-lc-your-key" \
  -H "Content-Type: application/json" \
  -d '{"model": "your-model.gguf", "messages": [{"role": "user", "content": "Hello"}]}'

This works with the OpenAI SDKs, LangChain, opencode, aider and Continue. Members can also chat with permitted models directly in the Console.

Step 5: Set up people and permissions

Create an organization, invite members and group them into departments. Each department is granted access to specific models, and each person gets personal sk-lc- API keys and usage statistics. That gives you an audit trail of who is using what, without Loadcore ever storing prompts or completions.

What to know before you start

Share the architecture with your security team early. It is simple to approve:

  • Inference traffic goes from your tools straight to the agent on your server.
  • Control traffic (heartbeats, download progress, instance status, key validation) goes to the Loadcore control plane.
  • The agent only needs an outbound connection. Your users reach it inside your own network.
  • To grow, add servers and run more instances of the models your teams use most.

For a large model that does not fit one GPU, Loadcore can split it across up to 8 servers on a local network using llama.cpp RPC. That is for fitting bigger models, not for speed. For more throughput, run the same model on several servers as separate instances. More on this in How to Manage a GPU Fleet for LLM Inference.

Next steps