Running Gemma 4 on Your Own GPU Servers
How to run Gemma 4 on your own GPU servers: real file sizes, hardware fit for the 31B and 26B A4B models, recommended settings and deployment steps.
Running Gemma 4 on your own GPU servers gives your company a capable, Apache 2.0 licensed model with a 262,144 token context window, and keeps every prompt on infrastructure you control. This article covers the two Gemma 4 models in the Loadcore library, what hardware they need and how to deploy them.
Two models, two trade-offs
Google's Gemma 4 family includes a dense model and a mixture-of-experts model. Both are instruction tuned, accept image input and are licensed under Apache 2.0.
| Gemma 4 31B | Gemma 4 26B A4B | |
|---|---|---|
| Architecture | 31B dense | 26B mixture of experts, 3.8B active |
| Context window | 262,144 tokens | 262,144 tokens |
| License | Apache 2.0 | Apache 2.0 |
| Strength | Largest Gemma 4, dense | Serves many more tokens per second |
Gemma 4 31B
This is the dense flagship of the family. Because every parameter is used for every token, it is the better pick when you want the most from a single model on a modest server.
Gemma 4 26B A4B
This model uses 128 experts plus one shared expert, with 8 active per token, which gives 3.8B active parameters out of 26B. Google describes it as running almost as fast as a 4B model. Its memory footprint is close to the 31B, but it is a good choice for high-traffic chat where many people use the same server.
Hardware fit
These are the GGUF file sizes from the Unsloth repositories on Hugging Face.
Gemma 4 31B
| Quantization | Weights |
|---|---|
| Q4_K_M | 18.3 GB |
| Q6_K | 25.2 GB |
| Q8_0 | 33.2 GB |
Gemma 4 26B A4B
| Quantization | Weights |
|---|---|
| MXFP4 | 16.6 GB |
| UD-Q4_K_XL | 17.0 GB |
| Q8_0 | 27.3 GB |
What this means in practice:
- A 48 GB card such as an L40S runs the 31B at Q8_0 with room for context.
- A 24 GB card runs the 31B at Q4_K_M with a shorter context.
- The 26B A4B at around 17 GB suits 24 to 48 GB cards.
Weights are not the whole story. Budget extra VRAM for the KV cache, which grows with context size and with the number of parallel requests. A long context window is only useful if you leave memory for it.
Vision input needs the separate mmproj file from the GGUF repository.
Which one should you choose?
- Choose 31B if you want the strongest Gemma 4 output and your usage is moderate.
- Choose 26B A4B if many employees will chat at the same time and responsiveness matters more.
If you have two servers, you can run both and let departments use whichever suits their work. Loadcore grants model access per department, so this is a configuration choice, not an engineering project.
Deploying Gemma 4 with Loadcore
- Install Loadcore Agent on a Linux GPU server (see Getting Started with Self-Hosted LLMs).
- In the Console, open Models and search for
unsloth/gemma-4-31B-it-GGUForunsloth/gemma-4-26B-A4B-it-GGUF. - Pick the quantization that fits the server and start the download. The agent saves it to
~/.loadcore/modelsand shows progress. - Start an instance with the context size you need and GPU layers set to offload the full model.
- Point your tools at
http://your-server:9090/v1with a personalsk-lc-key and the model file name.
Recommended settings
Google's recommended defaults from the model card apply to both models:
| Parameter | Value |
|---|---|
| temperature | 1.0 |
| top_p | 0.95 |
| top_k | 64 |
| repeat_penalty | 1.0 |
| min_p | 0.0 |
What to expect
Gemma 4 is a strong general-purpose model for drafting, summarizing, translation, internal Q&A and image understanding. For heavier reasoning and coding work, pair it with a larger model such as gpt-oss 120B on the same fleet. Test with your own real prompts: your workload is the only benchmark that counts.
Sizing for a team
Pick the quantization by the server you have, then leave headroom. On a 48 GB card the 31B at Q8_0 uses 33.2 GB for weights, which leaves roughly 15 GB for the KV cache and runtime. If many people chat in parallel or you need very long contexts, that headroom shrinks quickly, so either choose Q6_K (25.2 GB) or lower the context size when you start the instance. Start conservative, watch memory on the server, and raise the context only if people actually need it.
For more capacity, run the same Gemma 4 model on a second server as a separate instance. Loadcore does not balance load between them automatically, so you decide which teams or tools use which server. See How to Manage a GPU Fleet for LLM Inference.
Why run it yourself
With Gemma 4 on your own server, prompts and completions never reach Loadcore or a model provider. That is relevant for teams under the GDPR (see GDPR-compliant AI) and for regulated sectors such as healthcare and financial services.
To compare against hosted options, see the self-hosted ChatGPT alternative page, browse all models or read the docs.