--- title: "ClikaRT::nn::MoE" sidebar_label: "MoE" description: "Dense mixture-of-experts module: a router picks top-k experts per token, each expert runs its gated FFN, and the outputs combine under the router weights." --- class **Header:** `ClikaRT/nn/moe.h` **Inherits:** [`ClikaRT::nn::Module`](./Module.md) Dense mixture-of-experts module: a router picks top-k experts per token, each expert runs its gated FFN, and the outputs combine under the router weights. The expert weights live as stacked `[E, ...]` tensors; the optional trailing [`MoeOptions`](./MoeOptions.md) carries the routing variants (grouped routing, sparse-mixer, the clamped-SwiGLU knobs, per-expert combine scales). The module owns its weights: construct with `make(...)`, or declare shapes and bind a checkpoint via [`load_state_dict`](./Module.md#load_state_dict). The first forward packs the weight (or [`LoadOptions::pack_on_load`](./LoadOptions.md#pack_on_load) packs it at the end of the load); the pack is the one resident copy, and [`named_parameters()`](./Module.md#named_parameters) / [`state_dict()`](./Module.md#state_dict) read the weight back from it; `to(dtype)` restores, casts, and re-packs on the next forward. ```cpp auto moe = ClikaRT::nn::MoE::make(router_w, gate_up_w, down_w, opts); auto y = moe->forward(x); ``` ## Static member functions ### make() {#make}

`static` `std::shared_ptr` **`make`**(
    `std::int64_t` `num_experts`,
    `std::int64_t` `hidden_size`,
    `std::int64_t` `intermediate_size`,
    `std::int64_t` `top_k`,
    `bool` `gate_up_bias` `=` `false`,
    `bool` `down_bias` `=` `false`,
    [`DataType`](../../index.md#DataType) `dtype` `=` [`DataType::Float32`](../../index.md#DataType),
    [`Device`](../Device.md) `device` `=` [`Device::cpu`](../Device.md#cpu)`()`,
    [`MoeOptions`](./MoeOptions.md) `options` `=` `{}`
)

Defaults: `gate_up_bias` / `down_bias` = `false` (no bias slots), [`dtype`](../dtype/index.md) = `Float32`, [`device`](../device-namespace/index.md) = CPU, `options` = [`MoeOptions`](./MoeOptions.md)`{}` (each field's own default; see the struct). The routing variants ride the optional trailing `options`; the four counts are the whole required surface. **Throws** - `ClikaRT::Error`: as stated above.

Declared in `ClikaRT/nn/moe.h`, line 114

## Member functions ### \~MoE() {#MoE}

**`~MoE`**() `override`

Releases the module's packed expert weights.

Declared in `ClikaRT/nn/moe.h`, line 126

### set\_weights() {#set_weights}

`void` **`set_weights`**(
    [`Tensor`](../Tensor.md) `gate_up_experts`,
    [`Tensor`](../Tensor.md) `down_experts`,
    [`OptionalTensor`](../../index.md#OptionalTensor) `gate_up_bias` `=` `{}`,
    [`OptionalTensor`](../../index.md#OptionalTensor) `down_bias` `=` `{}`,
    [`OptionalTensor`](../../index.md#OptionalTensor) `gate_experts` `=` `{}`,
    [`OptionalTensor`](../../index.md#OptionalTensor) `gate_bias` `=` `{}`,
    [`OptionalTensor`](../../index.md#OptionalTensor) `e_score_correction_bias` `=` `{}`
)

Bind the declared slots positionally: `gate_up_experts` `[E, F·I, H]` and `down_experts` `[E, H, I]` (required), the declared per-expert biases, the Unfused `gate_experts` `[E, I, H]` (+ `gate_bias`), and the aux-free selection bias `[E]` (bindable even when not declared; it is routing data, not a declared parameter). The declared placement wins; shapes must match the declaration. Re-binding after a pack drops the pack; the next [`forward`](./MoE.md#forward) re-packs. Raises [`ClikaRT::Error`](../Error.md) on a geometry mismatch or a bias without its declared slot.

Declared in `ClikaRT/nn/moe.h`, line 140

### to\_impl(StreamOrDevice) {#to_impl}

`virtual` [`Result`](../Result.md)`` **`to_impl`**([`StreamOrDevice`](../StreamOrDevice.md) `where`) `override`

Move to a placement (the pack rebuilds on the target) or cast to a dtype (the raw experts restore from the pack, cast, and re-pack on the next forward).

Declared in `ClikaRT/nn/moe.h`, line 155

### to\_impl(DataType) {#to_impl-2}

`virtual` [`Result`](../Result.md)`` **`to_impl`**([`DataType`](../../index.md#DataType) `dtype`) `override`

Declared in `ClikaRT/nn/moe.h`, line 156

### forward() {#forward}

[`Tensor`](../Tensor.md) **`forward`**(
    [`Tensor`](../Tensor.md) `x`,
    [`Tensor`](../Tensor.md) `router_logits`,
    [`OptionalTensor`](../../index.md#OptionalTensor) `shared_output` `=` `{}`
) `const`

One fused [MoE](./MoE.md) step: `x` `[T, H]` (packed/ragged rows are native, no cu\_seqlens; [MoE](./MoE.md) is per-token) and the caller-computed `router_logits` `[T, E]` → `[T, H]`. The first call packs (once, thread-safe); every later call reuses the pack. Raises [`ClikaRT::Error`](../Error.md) on a shape/dtype mismatch or a still-fake slot. An optional `shared_output` (the shared-expert branch's `[T, H]` at `x`'s dtype, EXACT shape) folds into the routed combine (the fused routed+shared epilogue; no separate add pass).

Declared in `ClikaRT/nn/moe.h`, line 168

## Protected member functions ### initialize\_impl() {#initialize_impl}

`virtual` [`Result`](../Result.md)`` **`initialize_impl`**() `override`

The pack hook ([`Module::initialize_impl`](./Module.md#initialize_impl)): packs now (idempotent, thread-safe). Every declared slot must hold a real (loaded) tensor; a still-fake slot is a clean error. After the pack the expert weight slots read back from the pack, the single resident copy; biases stay bound.

Declared in `ClikaRT/nn/moe.h`, line 185