Skip to main content

Load quantized weights from a GGUF file

You have a block-quantized GGUF checkpoint and want to run it without inflating it to dense floats. ClikaRT loads the file as-is: io::load_gguf returns the tensors plus the file's metadata, quantized entries keep their packed bytes, and nn::QLinearWoQ runs the matmul off the packed form. The checkpoint's on-disk footprint is its in-memory footprint.

The programs below use SmolLM2-135M-Instruct (Apache-2.0, 145 MB in Q8_0), small enough to download in a minute. Any .gguf file works; nothing here depends on the architecture.

curl -LO "https://huggingface.co/bartowski/SmolLM2-135M-Instruct-GGUF/resolve/main/SmolLM2-135M-Instruct-Q8_0.gguf"

Each program is a complete main.cpp; build them like any bundle consumer (tutorial part 1 has the four-line CMake project). The C samples abbreviate the api-table bootstrap that part 1 shows in full.

Load the file and read its metadata​

io::load_gguf returns a GgufModel: the weights as a NamedTensors map and the file's metadata as one Json object. The loader takes the target device (CPU by default), and the tensor payloads are mmap-backed, so loading is cheap.

inspect_metadata.cpp
#include <cstdio>

#include "ClikaRT/clika_rt.h"

namespace io = ClikaRT::io;

int main() {
    const io::GgufModel model = io::load_gguf("SmolLM2-135M-Instruct-Q8_0.gguf");

    std::printf("tensors  %zu\n", model.tensors.size());
    std::printf("metadata %zu keys\n", model.metadata.size());
    for (const char* key : {"general.architecture", "general.size_label",
                            "llama.block_count", "llama.embedding_length"}) {
        std::printf("  %-24s = %s\n", key, model.metadata.at(key).dump(0).c_str());
    }
    return 0;
}
tensors  272
metadata 37 keys
  general.architecture     = "llama"
  general.size_label       = "135M"
  llama.block_count        = 30
  llama.embedding_length   = 576

The metadata carries everything the file knows about itself: architecture, hyperparameters, tokenizer configuration (tokenizer.chat_template included; the chat-template guide picks that up). at(key) raises ClikaRT::Error on a missing key; probe with contains when a key is optional.

What a quantized weight is in memory​

A quantized entry rides a plain Tensor whose element data is the packed block stream, exactly as it sits in the file. is_quantized() separates those from the dense entries (norms and embeddings stay floating point in most files). quantized_view names what the payload is: the packed bytes, the scheme, and the logical element shape they encode.

inspect_weights.cpp
#include <cstdio>
#include <string_view>

#include "ClikaRT/clika_rt.h"

using ClikaRT::QTensor;
using ClikaRT::Tensor;
namespace io = ClikaRT::io;

int main() {
    const io::GgufModel model = io::load_gguf("SmolLM2-135M-Instruct-Q8_0.gguf");

    int quantized = 0, dense = 0;
    model.tensors.for_each([&](std::string_view, const Tensor& t) {
        t.is_quantized() ? ++quantized : ++dense;
    });
    std::printf("%d quantized, %d dense\n", quantized, dense);

    const QTensor q = ClikaRT::quantized_view(model.tensors.get("blk.0.ffn_up.weight"));
    std::printf("scheme  %s\n", q.scheme.c_str());
    std::printf("logical [%lld, %lld]\n",
                (long long)q.logical_shape[0], (long long)q.logical_shape[1]);
    std::printf("payload %s\n", q.payload.to_string().c_str());

    const Tensor norm = model.tensors.get("blk.0.attn_norm.weight");
    std::printf("dense   %s\n", norm.to_string().c_str());
    return 0;
}
211 quantized, 61 dense
scheme  GGUF_Q8_0
logical [576, 1536]
payload Tensor(shape=[1536, 612], dtype=UInt8, device=CPU, numel=940032, quantized=true, mmap=checkpoint:SmolLM2-135M-Instruct-Q8_0.gguf+33749344, data=[232, 27, 234, 24, 66, 230, ...])
dense   Tensor(shape=[576], dtype=Float32, device=CPU, numel=576, mmap=checkpoint:SmolLM2-135M-Instruct-Q8_0.gguf+31866976, data=[0.01398, 0.0238, -0.01978, -0.03027, -0.01965, -0.03516, ...])

Two facts to keep. The logical shape is [in_features, out_features], the row-contiguous dimension first; that is the orientation every quantized consumer below expects. The payload is UInt8 [rows, row_bytes]: for Q8_0, each row of 576 elements packs into blocks of 32 (one fp16 scale plus 32 int8 codes each), 34 bytes per block.

Serve it packed with QLinearWoQ​

nn::QLinearWoQ is a Linear over a quantized weight. The weight stays packed for the module's lifetime; the first forward reshapes the payload once into the backend's kernel layout, and every later call runs the quantized-weight matmul off that. Nothing is ever materialized dense.

The lifecycle is the same as every weight-bearing nn module: make(in, out) declares the slots, set_weights binds the loaded tensor, forward runs.

serve_packed.cpp
#include <cstdio>
#include <memory>

#include "ClikaRT/clika_rt.h"

using ClikaRT::DataType;
using ClikaRT::nn::QLinearWoQ;
using ClikaRT::QTensor;
using ClikaRT::Tensor;
namespace io = ClikaRT::io;

int main() {
    const io::GgufModel model = io::load_gguf("SmolLM2-135M-Instruct-Q8_0.gguf");
    const QTensor w = ClikaRT::quantized_view(model.tensors.get("blk.0.ffn_up.weight"));
    const std::int64_t in = w.logical_shape[0], out = w.logical_shape[1];

    const std::shared_ptr<QLinearWoQ> ffn_up = QLinearWoQ::make(in, out);
    ffn_up->set_weights(w);

    const Tensor x = Tensor::full({1, in}, 0.01, DataType::Float32);
    const Tensor y = ffn_up->forward(x);  // first call packs, later calls reuse
    std::printf("y = %s\n", y.to_string().c_str());
    return 0;
}
y = Tensor(shape=[1, 1536], dtype=Float32, device=CPU, numel=1536, data=[0.03397, -0.02917, 0.0793, 0.03382, -0.02785, 0.005078, ...])

One orientation trap: QLinearWoQ consumes the [in, out] logical shape that load_gguf produces, while dense Linear takes the HuggingFace [out, in] layout. Bind the GGUF entry as-is; do not transpose. Moving the module (ffn_up->to(...)) re-packs the weight on the target device, and a dtype move is refused: the weight stays quantized at rest.

Inspect a weight by dequantizing​

ops::dequantize decodes a packed weight into a dense tensor, [out, in] row-major. It is the inspection and tooling path, not the serving path; use it to eyeball values or to check a conversion. The program decodes the same weight, checks the packed forward against the dense one, and prints what staying packed saves.

check_dense.cpp
#include <cmath>
#include <cstdio>
#include <memory>
#include <vector>

#include "ClikaRT/clika_rt.h"

using ClikaRT::DataType;
using ClikaRT::nn::QLinearWoQ;
using ClikaRT::QTensor;
using ClikaRT::Tensor;
namespace io = ClikaRT::io;
namespace ops = ClikaRT::ops;

int main() {
    const io::GgufModel model = io::load_gguf("SmolLM2-135M-Instruct-Q8_0.gguf");
    const QTensor w = ClikaRT::quantized_view(model.tensors.get("blk.0.ffn_up.weight"));
    const std::int64_t in = w.logical_shape[0], out = w.logical_shape[1];

    const Tensor dense = ops::dequantize(w);  // [out, in], the scheme's float target
    std::printf("dense  = %s\n", dense.to_string().c_str());

    // The packed path and the dense path compute the same values.
    const std::shared_ptr<QLinearWoQ> packed = QLinearWoQ::make(in, out);
    packed->set_weights(w);
    const Tensor x = Tensor::full({1, in}, 0.01, DataType::Float32);
    const std::vector<float> y  = packed->forward(x).reshape({-1}).item_as_vec<float>();
    const std::vector<float> yr = ops::linear(x, dense).reshape({-1}).item_as_vec<float>();
    float max_diff = 0.0F;
    for (std::size_t i = 0; i < y.size(); ++i)
        max_diff = std::max(max_diff, std::fabs(y[i] - yr[i]));
    std::printf("max |packed - dense| over %zu outputs: %g\n", y.size(), max_diff);

    std::printf("bytes: dense %lld, packed %lld (%.2fx)\n",
                (long long)(dense.numel() * 4), (long long)w.payload.numel(),
                (double)(dense.numel() * 4) / (double)w.payload.numel());
    return 0;
}
dense  = Tensor(shape=[1536, 576], dtype=Float32, device=CPU, numel=884736, data=[-0.08493, 0.09265, 0.2548, -0.1004, 0.1699, -0.193, ...])
max |packed - dense| over 1536 outputs: 7.15256e-07
bytes: dense 3538944, packed 940032 (3.76x)

The packed forward matches the dequantize-then-ops::linear reference to float rounding, at 3.76x fewer bytes for Q8_0 (Q4 and Q5 schemes save more).

When the bytes did not come from GGUF​

A packed payload from any other source gets the same treatment through make_quantized(payload, scheme, logical_shape): a UInt8 CPU tensor of packed block rows, the scheme's name ("GGUF_Q8_0", "MXFP4_E8M0"), and the element shape it encodes, row-contiguous dimension first. Checkpoints that ship MXFP4 as a split pair, 16 nibble-packed code bytes plus one E8M0 scale byte per 32-element group, go through make_quantized_mxfp4(blocks, scales, logical_shape), which weaves the pair into the packed row layout the runtime consumes. Either way the result is the same QTensor the programs above served.

You can now open any GGUF checkpoint, say what every entry is, and serve its weights at their packed size. The bundle's io example (chapter 02_gguf) inspects arbitrary files from the command line, and the runtime example wraps modules like QLinearWoQ into sessions and batching for serving.