Skip to main content

ClikaRT::nn::QMoEWoQ

class

Header: ClikaRT/nn/qmoe_woq.h
Inherits: ClikaRT::nn::Module

QMoEWoQ: a bound Mixture-of-Experts block over QUANTIZED expert weights (the weight-only-quantization serving path), exposed as an nn::Module leaf: the activation stays floating point and the stacked expert weights stay quantized at rest for the module's lifetime; each expert weight decodes one block at a time inside the GEMM dot, never a dense copy. This is the module that serves a block-quantized MoE checkpoint (e.g. MXFP4 gpt-oss) at its on-disk footprint; the dense-expert sibling is MoE (exactly the Linear vs QLinearWoQ split).

The lifecycle (uniform across every weight-bearing nn module):

  1. make(experts, hidden, intermediate, top_k, options, ...), the ONE constructor: declares the storage-free expert / bias slots.
  2. set_weights(...) or load_state_dict(...) binds them; a quantized payload lands on its slot as-is (the scheme travels with it, no cast).
  3. forward(x, router_logits) packs on first call (once, thread-safe), then serves. initialize() remains available as an optional warm-up.

The caller computes the per-token router logits itself and hands them to forward beside the activations; routing (MoeRouting), the gated activation, and the gate‖up layout (SwigluFusion) are configuration (MoeOptions, shared with MoE).

Weight-only-quantized mixture-of-experts module: the stacked [E, ...] expert weights rest quantized and decode inside the per-expert kernels, MoE's weight-only sibling (same routing surface and options).

The module owns its weights: construct with make(...), or declare shapes and bind a checkpoint via load_state_dict. After the first forward (or initialize()) the weight lives ONLY in the backend's packed form (one resident copy); to(dtype) restores, casts, and repacks on the next forward.

Static member functions​

make()​

static std::shared_ptr<QMoEWoQ> make(
    std::int64_t num_experts,
    std::int64_t hidden_size,
    std::int64_t intermediate_size,
    std::int64_t top_k,
    bool gate_up_bias = false,
    bool down_bias = false,
    DataType dtype = DataType::Float32,
    Device device = Device::cpu(),
    MoeOptions options = {}
)

Defaults: gate_up_bias / down_bias = false (no bias slots), dtype = Float32, device = CPU, options = MoeOptions{} (each field's own default; see the struct). The routing variants ride the optional trailing options; the four counts are the whole required surface.

Throws

  • ClikaRT::Error: as stated above.

Declared in ClikaRT/nn/qmoe_woq.h, line 73

Member functions​

~QMoEWoQ()​

~QMoEWoQ() override

Releases the module's packed expert weights.

Declared in ClikaRT/nn/qmoe_woq.h, line 85

set_weights()​

void set_weights(
    QTensor gate_up_experts,
    QTensor down_experts,
    OptionalTensor gate_up_bias = {},
    OptionalTensor down_bias = {},
    QTensor gate_experts = {},
    OptionalTensor gate_bias = {},
    OptionalTensor e_score_correction_bias = {}
)

Bind the declared slots positionally: the QUANTIZED gate_up_experts / down_experts stacks (required; e.g. from io::load_gguf, make_quantized, make_quantized_mxfp4), the declared floating-point per-expert biases, the Unfused quantized gate_experts (+ gate_bias), and the aux-free selection bias [E] (bindable even when not declared). Payload logical geometry must match the declaration. Re-binding after a pack drops the pack; the next forward re-packs. Raises ClikaRT::Error on a geometry mismatch or a bias without its declared slot.

Declared in ClikaRT/nn/qmoe_woq.h, line 100

initialize_impl()​

virtual Result<void> initialize_impl() override

Optional warm-up: run the first-forward pack NOW (idempotent, thread-safe). Every declared slot must hold a real (loaded) tensor; a still-fake slot is a clean error. After the pack the expert weight slots are released (the packed form is the single resident copy); biases stay enumerable.

Declared in ClikaRT/nn/qmoe_woq.h, line 117

to_impl(StreamOrDevice)​

virtual Result<void> to_impl(StreamOrDevice where) override

Move to a placement (the packed experts rebuild on the target). A dtype to is Unsupported: the experts stay quantized at rest; dequantize explicitly if a dense copy is wanted.

Declared in ClikaRT/nn/qmoe_woq.h, line 122

to_impl(DataType)​

virtual Result<void> to_impl(DataType dtype) override

Declared in ClikaRT/nn/qmoe_woq.h, line 123

forward()​

Tensor forward(
    Tensor x,
    Tensor router_logits,
    OptionalTensor shared_output = {}
) const

One fused MoE step: x [T, H] and the caller-computed router_logits [T, E] → [T, H] (dtype mirrors x). The first call packs (once, thread-safe); every later call reuses the pack. Raises ClikaRT::Error on a shape/dtype mismatch or a still-fake slot. An optional shared_output (the shared-expert branch's [T, H] at x's dtype, EXACT shape) folds into the routed combine (the fused routed+shared epilogue; no separate add pass).

Declared in ClikaRT/nn/qmoe_woq.h, line 134