ClikaRT::ops::moe

function

moe()

Tensor moe(
    Tensor input,
    Tensor router_logits,
    Tensor fc1_experts,
    Tensor fc2_experts,
    int64_t top_k,
    OptionalTensor fc1_bias = {},
    OptionalTensor fc2_bias = {},
    OptionalTensor fc3_experts = {},
    OptionalTensor fc3_bias = {},
    OptionalTensor e_score_correction_bias = {},
    OptionalTensor router_weights = {},
    std::optional<MoeRouting> routing_mode = std::nullopt,
    std::optional<bool> renormalize = std::nullopt,
    std::optional<int64_t> n_group = std::nullopt,
    std::optional<int64_t> topk_group = std::nullopt,
    std::optional<double> routed_scaling_factor = std::nullopt,
    std::optional<double> sparse_mixer_eps = std::nullopt,
    std::optional<bool> apply_router_weight_on_input = std::nullopt,
    std::optional<Activation> activation = std::nullopt,
    std::optional<SwigluFusion> swiglu_fusion = std::nullopt,
    std::optional<double> swiglu_alpha = std::nullopt,
    std::optional<double> swiglu_beta = std::nullopt,
    std::optional<double> swiglu_limit = std::nullopt,
    std::optional<GeluMode> gelu_mode = std::nullopt,
    OptionalTensor shared_output = {}
)

Fused Mixture-of-Experts layer: route, run the top-k experts, combine in one call, with no per-expert dispatch from the caller.

Per token, router_logits [T, E] select top_k experts under routing_mode; each selected expert applies its own MLP (fc1 [E, F·I, H] → activation → fc2 [E, H, I], with F = 2 for a gated activation, else 1, and an optional multiplicative fc3 [E, I, H] branch); the expert outputs combine under the routing weights.

input is [T, H]; the result is [T, H] at input's dtype. Per-expert biases ride fc1_bias / fc2_bias / fc3_bias; e_score_correction_bias and n_group / topk_group / routed_scaling_factor serve the group-limited routing families; router_weights feeds MoeRouting::PreComputed (caller-supplied combine weights); sparse_mixer_eps tunes MoeRouting::SparseMixer. The gated-activation scalars (swiglu_*, gelu_mode) carry their swiglu / geglu meanings; shared_output [T, H] folds a shared-expert branch into the final combine. A config argument left std::nullopt takes the runtime default (SoftmaxTopK routing, renormalized top-k weights).

Returns: the combined expert output [T, H].

Throws

ClikaRT::Error: (INVALID_ARGUMENT) when the expert-weight geometry disagrees with input / router_logits, or a routing config value is outside its family's range (code_name() carries the reason).

Declared in ClikaRT/compute/ops.h, line 2829

moe()​

moe()