Skip to main content

ClikaRT::ops::qk_layer_norm

function

qk_layer_norm()​

std::array<Tensor 3> qk_layer_norm(
    Tensor q,
    OptionalTensor k = {},
    OptionalTensor v = {},
    OptionalTensor w_q = {},
    OptionalTensor b_q = {},
    OptionalTensor w_k = {},
    OptionalTensor b_k = {},
    std::int64_t head_dim = 0,
    std::optional<double> eps = std::nullopt
)

Per-head LAYER norm over packed attention projections, the mean-subtracting sibling of qk_rms_norm.

Each contiguous head_dim run of q (and k, when present) is normalized as (x - mean) / sqrt(var + eps) * w + b; v passes through untouched.

Parameters

  • q: the query projection, [.., heads*head_dim].
  • k: optional key projection, same trailing structure.
  • v: optional value projection; passed through unchanged.
  • w_q: optional per-head gain for q, shape [head_dim].
  • b_q: optional per-head shift for q, shape [head_dim].
  • w_k: optional per-head gain for k, shape [head_dim].
  • b_k: optional per-head shift for k, shape [head_dim].
  • head_dim: the per-head run length; must divide the trailing extent.
  • eps: stability floor; std::nullopt selects 1e-5.

Returns: {q_normed, k_normed, v}; absent inputs come back absent.

Throws

  • ClikaRT::Error: (INVALID_ARGUMENT) when head_dim does not divide the trailing extent or an affine operand is mis-shaped.

Declared in ClikaRT/compute/ops.h, line 3565