Skip to main content

Run a specific GGUF quantization

A GGUF repo usually publishes one model in several quantizations, from a 2-bit file that fits a phone to an 8-bit one that wants a workstation. Modelverse treats those as weight options of one source: it never guesses between them, the option table with sizes prints before any download, and a selector names the one you want.

See the options​

Ask with --dry. A source with several weight options leads with the option table:

clika-modelverse info Qwen/Qwen3-4B-GGUF --dry
Qwen/Qwen3-4B-GGUF
  family=qwen  model_type=  architecture=qwen3
  variant: dense
  components: tokenizer=no image=no audio=no video=no
'Qwen/Qwen3-4B-GGUF' ships 5 weight options; pick one:

  option           files  size       selector
  Qwen3-4B-Q4_K_M  1      2.3 GiB    :Q4_K_M
  Qwen3-4B-Q5_0    1      2.6 GiB    :Q5_0
  Qwen3-4B-Q5_K_M  1      2.7 GiB    :Q5_K_M
  Qwen3-4B-Q6_K    1      3.1 GiB    :Q6_K
  Qwen3-4B-Q8_0    1      4.0 GiB    :Q8_0

pick by tag (":Q6_K" / --weights Q6_K; must match one option), by glob (":*Q8*"), by exact file (org/repo/<file>.gguf, a /blob|/resolve URL, or --weights <file>.gguf), or ":latest" for the hub default
  companions:
    .gitattributes                                      1.8 KiB
    LICENSE                                            11.3 KiB
    README.md                                           7.4 KiB
    params                                                270 B
  weights:
    Qwen3-4B-Q4_K_M.gguf                                2.3 GiB
    Qwen3-4B-Q5_0.gguf                                  2.6 GiB
    Qwen3-4B-Q5_K_M.gguf                                2.7 GiB
    Qwen3-4B-Q6_K.gguf                                  3.1 GiB
    Qwen3-4B-Q8_0.gguf                                  4.0 GiB
  total: 14.7 GiB  (weights 14.7 GiB, 9 files)
  (dry; nothing downloaded)

Running the bare source refuses with the same table and exit code 2; the refusal is the answer to "what is there", not an obstacle.

Select one​

A selector rides the source after a colon, and the same grammar works on every command the model provides:

clika-modelverse Qwen/Qwen3-4B-GGUF:Q6_K prompt "The capital of France is"

The refusal's own closing line is the whole grammar; the four forms, most specific last:

  • A tag, Qwen/Qwen3-4B-GGUF:Q6_K. An exact option name resolves outright; otherwise a case-insensitive contains-match runs over the option names and each option's first file name. The everyday form.
  • A glob, Qwen/Qwen3-4B-GGUF:*Q4*. Must match exactly one option; an ambiguous selector refuses with every name (file) candidate listed.
  • :latest, the repo's default option, an explicit opt-in rather than a silent fallback.
  • An exact file, Qwen/Qwen3-4B-GGUF/Qwen3-4B-Q6_K.gguf, a pasted Hugging Face Hub blob/resolve URL, or a unique trailing part of the file name. Unambiguous, the right form for scripts that must never drift when the repo adds options.

fetch takes the same selection, either in the source or as --weights Q6_K, and prints the selected weight file's local path as its payload:

WEIGHTS=$(clika-modelverse fetch Qwen/Qwen3-4B-GGUF --weights Q6_K)

Pick the compute dtype​

On a quantized checkpoint, --dtype (float16, bfloat16, float32) selects the compute dtype: the precision every quantized weight decodes to and the activations and KV cache run at, behind the generative, embedding and reranker loads. The option pins the whole tree, and the token table of a quantized checkpoint stays quantized in memory, decoding each gathered row at the selected dtype, so a multi-gigabyte f32 table is never materialized. Absent, the checkpoint's own dtype rules. On a dense checkpoint the option is the weight cast applied at load: a llama checkpoint takes float16, bfloat16 or float32, an object-detection checkpoint float16 or bfloat16; a dense checkpoint of any other family refuses the option by name.

clika-modelverse Qwen/Qwen3-4B-GGUF:Q6_K prompt "The capital of France is" --dtype float16

What counts as one option​

An option is a group, not always a file: a multi-part split (<stem>-00001-of-00009.gguf) is one option named by its stem (the table's files column counts its parts), and a subdirectory holding one quantization's parts is one option named by the subdirectory. Splits are listed with a (split; not served yet) note and are selectable by name, but loading one refuses precisely; the loader serves single-file GGUF. Sidecar files (mmproj, imatrix) are excluded from the options and the table says how many it left out.

Which quantization fits which device is a sizing question; Model requirements carries the figures. The loading path underneath (block-quantized weights decoded inside the kernels, served at their on-disk footprint) is ClikaRT's, documented in its GGUF guide on /clikart.