Local LLMs come in a confusing array of file variants. The same model, say a hypothetical 14-billion-parameter model, will appear on a community quantizer's repository as model-14b-Q4_K_M.gguf, model-14b-Q5_K_S.gguf, model-14b-Q6_K.gguf, model-14b-IQ3_XXS.gguf, and a dozen others. They represent the same base architecture with differently approximated weights. The compression is what we call quantization, and the names tell you exactly what kind of compression was applied. Once you can read them, you stop guessing.
A neural network is a very long list of numbers. Those numbers are weights, and during training they were stored as 32-bit or 16-bit floating-point values. A 7-billion-parameter model in 16-bit is 14 GB of weights. A 70-billion-parameter model in 16-bit is 140 GB. A 14 GB weight set can fit some consumer GPUs but needs extra space at runtime; 140 GB exceeds a typical single consumer GPU, and even system RAM struggles with the bigger ones. So we compress.
Quantization is a fancy word for "store each weight in fewer bits." If a weight that used 16 bits can be approximated with 4 bits, the raw weight storage would shrink to one quarter before scales, mixed-precision tensors, and metadata are counted. The model still runs; it just runs from a slightly less precise version of itself. The art is in choosing where to lose precision so that the loss does not break the model's behavior.
Pure 4-bit storage would be the simplest scheme: take every weight, pick the closest representable value out of 16 possibilities, store the index. That works, but it loses too much. Modern formats are smarter. They group weights into small blocks (typically 32 values per block), and within each block they store a shared scale factor and sometimes an offset. So instead of "this weight is the integer 5 out of 0..15," it is "this weight is 5 out of 0..15, multiplied by the block's scale of 0.0042, with the block's offset added." That tiny per-block calibration brings most of the precision back.
The K family uses block and super-block quantization, introduced in llama.cpp's June 2023 k-quants change. Stored scales and other metadata make the effective bits per weight higher than the nominal number in the name. Mixed presets can also keep selected tensors at higher precision.
The names follow a pattern. The first part is the storage scheme, the second part is the bit count, and the suffix is the size variant. Let me break down the most common ones you will actually see.
Q2_K through Q6_K are the familiar k-quant families for model weights. The commonly distributed eight-bit weight format is Q8_0. Q8_K is used for intermediate calculations in the original k-quants implementation; it is not the usual eight-bit GGUF download label. Higher precision generally reduces quantization error, but task quality still needs checking.
The suffix _S, _M, or _L is the size variant within a bit budget. They control how the format spreads its bit budget across different parts of the model. _S is the smallest and most aggressive. _L is the largest and most generous. _M sits in the middle.
Names such as IQ2_XXS, IQ3_XXS, and IQ4_XS belong to a different family of low-bit formats. Several use calibration information to weight quantization error; that is more precise than saying each important weight simply receives more bits. IQ4_NL uses a non-linear quantization scheme and is not a universal example of importance-matrix calibration. The project's tensor-format reference explains the layouts. A suffix describes a particular preset, not a quality guarantee shared across families.
A model with fourteen billion parameters gives a rough sense of size: about 28 GB of weights at 16 bits, around 15 GB with Q8_0, and around 8–9 GB with Q4_K_M. These are decimal GB estimates, not exact files or runtime memory requirements. Different architectures, unquantized tensors, and metadata change the totals. The context cache and runtime buffers need additional memory.
At lower bit counts, compression becomes more aggressive and errors may become more visible. The effect depends on the model, calibration, and task. An IQ format can offer a useful size/quality trade-off, but this article does not provide a benchmark proving that one preset always beats another. Test the actual model on work you care about.
Hardware decides what you can run. The first constraint is memory. The model must fit in your GPU's VRAM (or, if you do CPU-only inference, in system RAM with some headroom for the OS). If your GPU has 12 GB of VRAM and the model file is 14 GB, you cannot load it; you have to drop to a smaller quantization or split it across CPU and GPU, which slows it down a lot.
The second constraint is bandwidth. Lower-bit weights mean fewer bytes to move, which can help generation when memory bandwidth is the bottleneck. Dequantization cost, kernel support, batching, and hardware also matter. There is no general 1.5–2× speedup guaranteed by moving from Q8_0 to Q4_K_M; measure both on your runtime if speed is the deciding factor.
I treat quantization as a hardware-tier decision. Pick the highest bit count that fits comfortably in memory and gives you tolerable speed; that is your sweet spot. For most modern consumer GPUs with 12 to 24 GB, Q4_K_M through Q6_K is the practical band. For machines with very tight memory budgets, the IQ formats give you a smarter way down.
A few common labels, condensed into a table for quick reference. Sizes shown are approximate for a 14-billion-parameter model.
| Name | Nominal bits/weight* | Approx. size (14B) | Use when |
|---|---|---|---|
F16 / BF16 | 16 | ~28 GB | You have the memory and want the original. |
Q8_0 | 8 nominal | ~15 GB | Lower compression error, at roughly half the F16 weight size. |
Q6_K | 6 | ~11 GB | More precision when runtime memory allows. |
Q5_K_M | 5 | ~9.5 GB | More precision than Q4_K_M when memory allows; compare on your task. |
Q4_K_M | 4 | ~8.5 GB | A common starting point if weights and runtime memory fit. |
Q4_K_S | 4 | ~8 GB | One step smaller than Q4_K_M, slightly more quality loss. |
Q3_K_M | 3 | ~6.5 GB | Last resort on tight memory. |
IQ4_XS | ~4.25 | ~7.7 GB | Compare size and task quality with Q4_K presets. |
IQ3_XXS | ~3.06 | ~5.5 GB | A smaller file when memory is tight; validate task quality. |
IQ2_XXS | ~2.06 | ~4 GB | Aggressive compression with a larger risk of quality loss. |
For other model sizes, weight storage scales roughly with parameter count. Estimate decimal GB as parameters in billions × effective bits per weight / 8. Fourteen billion weights at an effective 4.8 bits take about 8.4 GB before remaining file overhead. The table's K-format numbers are nominal bit counts; scales, metadata, and mixed tensors raise the effective rate. Download size is still the better guide for a particular file.
Leave memory for the runtime. The key/value cache, which retains attention state for the context, can need substantially more than a couple of GB at long contexts. Model architecture, cache precision, and parallel requests all affect it. A 16 GB GPU may accommodate a 14B model in Q5 or Q6 at a suitable context length; a smaller GPU may need stronger compression, partial CPU offload, or a smaller model. Check the actual allocation instead of treating the file size as the whole budget.
Quantization gives you a hardware trade-off rather than a universal quality grade. The same Q4_K_M file can be perfect for your hardware and task, wasteful, or insufficient. The useful setting is the one that matches your available memory and required speed. Once you can read the names, you can often pick it on the first try instead of downloading three files to find out.