Skip to main content
NEURONPLANT
NP-VRAM/GPU memory

How much VRAM does this context actually take?

Weights, KV cache, and runtime, kept apart. KV follows the model's layers and KV heads, including output tokens and concurrent sequences. A planning estimate, not a measured runtime.

Mode
Model

36 layers · 32 Q heads · 8 KV heads · head 128 · 8.2B

Weight precision
KV cache precision
Quantization group sizeweights per scale group

AWQ adds one FP16 scale per group. A smaller group costs more VRAM. 128 is the common pack, not a measured file.

Hardware

96 GB GDDR7 ECC · 600 W. 96 GB GDDR7 ECC, 600 W, 1,792 GB/s. Workstation Edition is one card. The Server Edition is the same memory bin in a chassis; this planner allows 1, 2, 4, or 8. Eight cards are not one address space and not a DGX. Published MIG: 4×24 GB, 2×48 GB, or 1×96 GB.

How many of these GPUs

Inference engine

Packs copies until the card is full. It does not model an engine claiming the rest of the GPU.

Context length

Trained window is 32,768 tokens. A longer length is not this model, including a documented extension.

Expected output tokens

Held in the KV cache on top of the prompt. Prompt plus output cannot pass the trained window.

Concurrent sequences
Safety margin