Skip to main content

ClikaRT::ops::add_rms_norm

function

add_rms_norm()​

std::array<Tensor 2> add_rms_norm(
    Tensor x,
    OptionalTensor residual = {},
    OptionalTensor residual2 = {},
    OptionalTensor post_residual = {},
    ClikaRT::Span<const std::int64_t> normalized_shape = {},
    OptionalTensor skip_bias = {},
    OptionalTensor weight = {},
    OptionalTensor bias = {},
    std::optional<double> eps = std::nullopt,
    std::optional<Activation> activation = std::nullopt
)

Fused residual-add + normalization, on either side of the norm. Which data flow runs is inferred from which addends you supply; there is no mode flag:

  • residual only → {ACT(norm(x + residual)·w + b), x + residual}
  • post_residual only → {ACT(norm(x)·w + b) + post_residual, UNDEFINED}
  • both → {ACT(norm(x + residual)·w + b) + post_residual, x + residual}
  • neither → raises (that is a plain rms_norm/layer_norm)

The second result is the PRE-norm sum, the residual stream the next sub-layer reads. Supplying only post_residual forms no such sum, so that element comes back UNDEFINED: read only the first one in that flow.

The activation applies to the norm result, before post_residual is added; it is the norm's epilogue, not the sum's. Gated activations (SwiGlu / GeGlu / ReGlu) narrow their input and are rejected.

Returns: a 2-element array {normalized, pre-norm sum}. residual2 is an optional second PRE-norm full-shape addend (a parallel residual stream, e.g. a gated per-layer delta): the pre-norm sum is x + residual (+ residual2) (+ skip_bias). It follows residual (the pre-norm addends are adjacent) and precedes post_residual.

Declared in ClikaRT/compute/ops.h, line 3513