---
title: Run a specific GGUF quantization
description: Pick one weight option of a repo that ships many, by tag, glob or exact file, and see the size table before anything downloads.
---

A GGUF repo usually publishes one model in several quantizations, from a 2-bit file that fits a phone to an 8-bit one that wants a workstation. Modelverse treats those as weight options of one source: it never guesses between them, the option table with sizes prints before any download, and a selector names the one you want.

## See the options

Ask with `--dry`. A source with several weight options leads with the option table:

```bash
clikart-cli info Qwen/Qwen3-4B-GGUF --dry
```

```text
Qwen/Qwen3-4B-GGUF
  family=qwen  model_type=  architecture=qwen3
  variant: dense
  components: tokenizer=no image=no audio=no video=no
  license page: https://github.com/QwenLM/Qwen/blob/main/Tongyi%20Qianwen%20LICENSE%20AGREEMENT
  model: Qwen/Qwen3-4B-GGUF  provider: Qwen  source: https://huggingface.co/Qwen/Qwen3-4B-GGUF
license: apache-2.0  https://huggingface.co/Qwen/Qwen3-4B-GGUF/blob/main/LICENSE
'Qwen/Qwen3-4B-GGUF' ships 5 weight options; pick one:

  option           files  size       selector
  Qwen3-4B-Q4_K_M  1      2.3 GiB    :Q4_K_M
  Qwen3-4B-Q5_0    1      2.6 GiB    :Q5_0
  Qwen3-4B-Q5_K_M  1      2.7 GiB    :Q5_K_M
  Qwen3-4B-Q6_K    1      3.1 GiB    :Q6_K
  Qwen3-4B-Q8_0    1      4.0 GiB    :Q8_0

pick by tag (":Q6_K" / --weights Q6_K; must match one option), by glob (":*Q8*"), by exact file (org/repo/<file>.gguf, a /blob|/resolve URL, or --weights <file>.gguf), or ":latest" for the hub default
  total: nothing is fetched until one option is selected
  repository: 14.7 GiB in 9 files; the rest is not fetched (other weight formats, files the fetch never takes)
  (dry; nothing downloaded)
```

Running the bare source refuses with the same table and exit code 2; the refusal is the answer to "what is there", not an obstacle.

## Select one

A selector rides the source after a colon, and the same grammar works on every command the model provides:

```bash
clikart-cli Qwen/Qwen3-4B-GGUF:Q6_K prompt "The capital of France is"
```

The refusal's own closing line is the whole grammar; the four forms, most specific last:

- **A tag**, `Qwen/Qwen3-4B-GGUF:Q6_K`. An exact option name resolves outright; otherwise a case-insensitive contains-match runs over the option names and each option's first file name. The everyday form.
- **A glob**, `Qwen/Qwen3-4B-GGUF:*Q4*`. Must match exactly one option; an ambiguous selector refuses with every `name (file)` candidate listed.
- **`:latest`**, the repo's default option, an explicit opt-in rather than a silent fallback.
- **An exact file**, `Qwen/Qwen3-4B-GGUF/Qwen3-4B-Q6_K.gguf`, a pasted Hugging Face Hub `blob`/`resolve` URL, or a unique trailing part of the file name. Unambiguous, the right form for scripts that must never drift when the repo adds options.

`fetch` takes the same selection, either in the source or as `--weights Q6_K`, and prints the selected weight file's local path as its payload:

```bash
WEIGHTS=$(clikart-cli fetch Qwen/Qwen3-4B-GGUF --weights Q6_K)
```

## Pick the compute dtype

On a quantized checkpoint, `--dtype` (`float16`, `bfloat16`, `float32`) selects the compute dtype: the precision every quantized weight decodes to and the activations and KV cache run at, behind the generative, embedding and reranker loads. The option pins the whole tree, and the token table of a quantized checkpoint stays quantized in memory, decoding each gathered row at the selected dtype, so a multi-gigabyte f32 table is never materialized. Absent, the checkpoint's own dtype rules. On a dense checkpoint the option is the weight cast applied at load: a llama checkpoint takes `float16`, `bfloat16` or `float32`, an object-detection checkpoint `float16` or `bfloat16`; a dense checkpoint of any other family refuses the option by name.

```bash
clikart-cli Qwen/Qwen3-4B-GGUF:Q6_K prompt "The capital of France is" --dtype float16
```

## What counts as one option

An option is a group, not always a file: a multi-part split (`<stem>-00001-of-00009.gguf`) is one option named by its stem (the table's `files` column counts its parts), and a subdirectory holding one quantization's parts is one option named by the subdirectory. Splits are listed with a `(split; not served yet)` note and are selectable by name, but loading one refuses precisely; the loader serves single-file GGUF. Sidecar files (mmproj, imatrix) are excluded from the options and the table says how many it left out.

Which quantization fits which device is a sizing question; [Model requirements](../model-requirements.md) carries the figures. The loading path underneath (block-quantized weights decoded inside the kernels, served at their on-disk footprint) is ClikaRT's, documented in its GGUF guide on [/clikart](/clikart).