All models
NVIDIA
Nemotron 3.5 Lightning 30B A3B
NVIDIA's hybrid Mamba and mixture-of-experts reasoning model. 30B total, 3B active, built for throughput.
moemambareasoningfast
Overview
Nemotron 3.5 Lightning interleaves Mamba-2 and MoE layers with a few attention layers. It has 30B total and 3B active parameters and a 262,144 token context. English and code are its primary languages.
Hardware fit
The ggml-org GGUF provides Q8_0 at 35.8 GB, which fits a 48 GB card. Community repos also publish smaller quantizations.
| Quantization | Weights |
|---|---|
| Q8_0 | 35.8 GB |
| BF16 | 67.3 GB |
Sizes are the GGUF file sizes from ggml-org/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF. Budget extra VRAM for the KV cache, which grows with context size and parallel requests.
Deploying on Loadcore
- In the console, open Models and search for
ggml-org/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF. - Pick the quantization that fits the target server and start the download.
- Start an instance with the context size you need and GPU layers set to offload the full model.
- Point your tools at
http://your-server:9090/v1with your personal key.
Recommended settings
| Parameter | Value |
|---|---|
| temperature | 1.0 |
| top_p | 0.95 |
| top_k | 0 |
| repeat_penalty | 1.0 |
| min_p | 0.0 |
NVIDIA recommends temperature 1.0 and top_p 0.95.