---
title: Load quantized weights from a GGUF file
description: Read a block-quantized checkpoint, keep the weights packed, and serve them through QLinearWoQ at their on-disk footprint.
---

You have a block-quantized GGUF checkpoint and want to run it without inflating it to dense floats. ClikaRT loads the file as-is: `io::load_gguf` returns the tensors plus the file's metadata, quantized entries keep their packed bytes, and `nn::QLinearWoQ` runs the matmul off the packed form. The checkpoint's on-disk footprint is its in-memory footprint.

The programs below use [SmolLM2-135M-Instruct](https://huggingface.co/bartowski/SmolLM2-135M-Instruct-GGUF) (Apache-2.0, 145 MB in Q8_0), small enough to download in a minute. Any `.gguf` file works; nothing here depends on the architecture.

```bash
curl -LO "https://huggingface.co/bartowski/SmolLM2-135M-Instruct-GGUF/resolve/main/SmolLM2-135M-Instruct-Q8_0.gguf"
```

Each program is a complete `main.cpp`; build them like any bundle consumer ([tutorial part 1](../getting-started/first-program/01-your-first-program.mdx) has the four-line CMake project).

## Load the file and read its metadata

`io::load_gguf` returns a `GgufModel`: the weights as a `NamedTensors` map and the file's metadata as one `Json` object. The loader takes the target device (CPU by default), and the tensor payloads are mmap-backed, so loading is cheap.

<LangTabs>

<LangTab value="cpp">

```cpp title="inspect_metadata.cpp"
int main() {
    const io::GgufModel model = io::load_gguf(howto::gguf_fixture());

    std::printf("tensors  %zu\n", model.tensors.size());
    std::printf("metadata %zu keys\n", model.metadata.size());
    for (const char* key : {"general.architecture", "general.size_label",
                            "llama.block_count", "llama.embedding_length"}) {
        std::printf("  %-24s = %s\n", key, model.metadata.at(key).dump(0).c_str());
    }
    return 0;
}
```

</LangTab>

<LangTab value="python">

```python title="inspect_metadata.py"
def main() -> None:
    tensors, metadata_json = crt.io.load_gguf(gguf_path())
    metadata = json.loads(metadata_json)

    print(f"tensors  {len(tensors)}")
    print(f"metadata {len(metadata)} keys")
    for key in ("general.architecture", "general.size_label",
                "llama.block_count", "llama.embedding_length"):
        print(f"  {key:<24} = {json.dumps(metadata[key])}")


if __name__ == "__main__":
    main()
```

</LangTab>

</LangTabs>

```text
tensors  272
metadata 37 keys
  general.architecture     = "llama"
  general.size_label       = "135M"
  llama.block_count        = 30
  llama.embedding_length   = 576
```

The metadata carries everything the file knows about itself: architecture, hyperparameters, tokenizer configuration (`tokenizer.chat_template` included; [the chat-template guide](tokenize-and-chat-templates.mdx) picks that up). `at(key)` raises `ClikaRT::Error` on a missing key; probe with `contains` when a key is optional.

## What a quantized weight is in memory

A quantized entry rides a plain `Tensor` whose element data is the packed block stream, exactly as it sits in the file. `is_quantized()` separates those from the dense entries (norms and embeddings stay floating point in most files). `quantized_view` names what the payload is: the packed bytes, the scheme, and the logical element shape they encode.

<LangTabs>

<LangTab value="cpp">

```cpp title="inspect_weights.cpp"
int main() {
    const io::GgufModel model = io::load_gguf(howto::gguf_fixture());

    int quantized = 0, dense = 0;
    model.tensors.for_each([&](std::string_view, const Tensor& t) {
        if (t.is_quantized()) ++quantized; else ++dense;
    });
    std::printf("%d quantized, %d dense\n", quantized, dense);

    const QTensor q = ClikaRT::quantized_view(model.tensors.get("blk.0.ffn_up.weight"));
    std::printf("scheme  %s\n", q.scheme.c_str());
    std::printf("logical [%lld, %lld]\n",
                static_cast<long long>(q.logical_shape[0]), static_cast<long long>(q.logical_shape[1]));
    std::printf("payload %s\n", q.payload.to_string().c_str());

    const Tensor norm = model.tensors.get("blk.0.attn_norm.weight");
    std::printf("dense   %s\n", norm.to_string().c_str());
    return 0;
}
```

</LangTab>

<LangTab value="python">

```python title="inspect_weights.py"
def main() -> None:
    tensors, _ = crt.io.load_gguf(gguf_path())

    quantized = sum(1 for t in tensors.values() if t.is_quantized)
    print(f"{quantized} quantized, {len(tensors) - quantized} dense")

    q = crt.quantized_view(tensors["blk.0.ffn_up.weight"])
    print(f"scheme  {q.scheme}")
    print(f"logical [{q.logical_shape[0]}, {q.logical_shape[1]}]")
    print(f"payload {q.payload}")

    print(f"dense   {tensors['blk.0.attn_norm.weight']}")


if __name__ == "__main__":
    main()
```

</LangTab>

</LangTabs>

```text
211 quantized, 61 dense
scheme  GGUF_Q8_0
logical [576, 1536]
payload Tensor(shape=[1536, 612], dtype=UInt8, device=CPU, numel=940032, quantized=true, mmap=checkpoint:SmolLM2-135M-Instruct-Q8_0.gguf+33749344, data=[232, 27, 234, 24, 66, 230, ...])
dense   Tensor(shape=[576], dtype=Float32, device=CPU, numel=576, mmap=checkpoint:SmolLM2-135M-Instruct-Q8_0.gguf+31866976, data=[0.01398, 0.0238, -0.01978, -0.03027, -0.01965, -0.03516, ...])
```

Two facts to keep. The **logical shape is `[in_features, out_features]`**, the row-contiguous dimension first; that is the orientation every quantized consumer below expects. The **payload is UInt8 `[rows, row_bytes]`**: for Q8_0, each row of 576 elements packs into blocks of 32 (one fp16 scale plus 32 int8 codes each), 34 bytes per block.

## Serve it packed with QLinearWoQ

`nn::QLinearWoQ` is a `Linear` over a quantized weight. The weight stays packed for the module's lifetime; the first `forward` reshapes the payload once into the backend's kernel layout, and every later call runs the quantized-weight matmul off that. Nothing is ever materialized dense.

The lifecycle is the same as every weight-bearing `nn` module: `make(in, out)` declares the slots, `set_weights` binds the loaded tensor, `forward` runs.

<LangTabs>

<LangTab value="cpp">

```cpp title="serve_packed.cpp"
int main() {
    const io::GgufModel model = io::load_gguf(howto::gguf_fixture());
    const QTensor w = ClikaRT::quantized_view(model.tensors.get("blk.0.ffn_up.weight"));
    const std::int64_t in = w.logical_shape[0], out = w.logical_shape[1];

    const std::shared_ptr<QLinearWoQ> ffn_up = QLinearWoQ::make(in, out);
    ffn_up->set_weights(w);

    const Tensor x = Tensor::full({1, in}, 0.01, DataType::Float32);
    const Tensor y = ffn_up->forward(x);  // first call packs, later calls reuse
    std::printf("y = %s\n", y.to_string().c_str());
    return 0;
}
```

</LangTab>

<LangTab value="python">

```python title="run_quantized.py"
def main() -> None:
    tensors, _ = crt.io.load_gguf(gguf_path())
    q = crt.quantized_view(tensors["blk.0.ffn_up.weight"])
    in_f, out_f = q.logical_shape

    # Dequantize to the scheme's float target and run dense math on it.
    # Serving the matmul off the PACKED form (no dense copy at rest) is
    # done through the quantized modules of the C++ API; the C++ tab
    # shows it.
    w = crt.dequantize(q)
    x = crt.full((1, in_f), 0.01)
    y = F.linear(x, w)
    print(f"y = {y}")


if __name__ == "__main__":
    main()
```

</LangTab>

</LangTabs>

```text
y = Tensor(shape=[1, 1536], dtype=Float32, device=CPU, numel=1536, data=[0.03397, -0.02917, 0.0793, 0.03382, -0.02785, 0.005078, ...])
```

One orientation trap: `QLinearWoQ` consumes the `[in, out]` logical shape that `load_gguf` produces, while dense `Linear` takes the HuggingFace `[out, in]` layout. Bind the GGUF entry as-is; do not transpose. Moving the module (`ffn_up->to(...)`) re-packs the weight on the target device, and a dtype move is refused: the weight stays quantized at rest.

## Inspect a weight by dequantizing

`ops::dequantize` decodes a packed weight into a dense tensor, `[out, in]` row-major. It is the inspection and tooling path, not the serving path; use it to eyeball values or to check a conversion. The program decodes the same weight, checks the packed forward against the dense one, and prints what staying packed saves.

<LangTabs>

<LangTab value="cpp">

```cpp title="check_dense.cpp"
int main() {
    const io::GgufModel model = io::load_gguf(howto::gguf_fixture());
    const QTensor w = ClikaRT::quantized_view(model.tensors.get("blk.0.ffn_up.weight"));
    const std::int64_t in = w.logical_shape[0], out = w.logical_shape[1];

    const Tensor dense = ops::dequantize(w);  // [out, in], the scheme's float target
    std::printf("dense  = %s\n", dense.to_string().c_str());

    // The packed path and the dense path compute the same values.
    const std::shared_ptr<QLinearWoQ> packed = QLinearWoQ::make(in, out);
    packed->set_weights(w);
    const Tensor x = Tensor::full({1, in}, 0.01, DataType::Float32);
    const Tensor y  = packed->forward(x);
    const Tensor yr = ops::linear(x, dense);
    std::printf("max |packed - dense| over %lld outputs: %g\n",
                static_cast<long long>(y.numel()),
                ops::amax(ops::abs(ops::sub(y, yr))).item<float>());

    std::printf("bytes: dense %zu, packed %zu (%.2fx)\n", dense.nbytes(), w.payload.nbytes(),
                static_cast<double>(dense.nbytes()) / static_cast<double>(w.payload.nbytes()));
    return 0;
}
```

</LangTab>

<LangTab value="python">

```python title="check_footprint.py"
def main() -> None:
    tensors, _ = crt.io.load_gguf(gguf_path())
    q = crt.quantized_view(tensors["blk.0.ffn_up.weight"])

    dense = crt.dequantize(q)  # [out, in], the scheme's float target
    print(f"dense  = {dense}")

    # The packed payload is the checkpoint's own byte stream; inflating it
    # to dense floats shows what staying packed saves. (The packed-vs-dense
    # VALUE check runs where the packed matmul lives: the C++ tab.)
    dense_bytes = dense.numpy().size * 4
    packed_bytes = q.payload.numpy().size
    print(f"bytes: dense {dense_bytes}, packed {packed_bytes} "
          f"({dense_bytes / packed_bytes:.2f}x)")


if __name__ == "__main__":
    main()
```

</LangTab>

</LangTabs>

```text
dense  = Tensor(shape=[1536, 576], dtype=Float32, device=CPU, numel=884736, data=[-0.08493, 0.09265, 0.2548, -0.1004, 0.1699, -0.193, ...])
max |packed - dense| over 1536 outputs: 7.18981e-07
bytes: dense 3538944, packed 940032 (3.76x)
```

The packed forward matches the dequantize-then-`ops::linear` reference to float rounding, at 3.76x fewer bytes for Q8_0 (Q4 and Q5 schemes save more).

## When the bytes did not come from GGUF

A packed payload from any other source gets the same treatment through `make_quantized(payload, scheme, logical_shape)`: a UInt8 CPU tensor of packed block rows, the scheme's name (`"GGUF_Q8_0"`, `"MXFP4_E8M0"`), and the element shape it encodes, row-contiguous dimension first. Checkpoints that ship MXFP4 as a split pair, 16 nibble-packed code bytes plus one E8M0 scale byte per 32-element group, go through `make_quantized_mxfp4(blocks, scales, logical_shape)`, which weaves the pair into the packed row layout the runtime consumes. Either way the result is the same `QTensor` the programs above served.

You can now open any GGUF checkpoint, say what every entry is, and serve its weights at their packed size. The bundle's `io` example (chapter `02_gguf`) inspects arbitrary files from the command line, and the [runtime example](../examples.md) wraps modules like `QLinearWoQ` into sessions and batching for serving.