Skip to main content

ClikaRT::ops::qk_layer_norm

function

qk_layer_norm()​

std::array<Tensor 3> qk_layer_norm(
    Tensor query,
    OptionalTensor key = {},
    OptionalTensor value = {},
    OptionalTensor query_weight = {},
    OptionalTensor query_bias = {},
    OptionalTensor key_weight = {},
    OptionalTensor key_bias = {},
    std::int64_t head_dim = 0,
    std::optional<double> eps = std::nullopt
)

Per-head LAYER norm over packed attention projections, the mean-subtracting sibling of qk_rms_norm.

Each contiguous head_dim run of query (and key, when present) is normalized as (x - mean) / sqrt(var + eps) * w + b; value passes through untouched.

Parameters

  • query: the query projection, [.., heads*head_dim].
  • key: optional key projection, same trailing structure.
  • value: optional value projection; passed through unchanged.
  • query_weight: optional per-head gain for query, shape [head_dim].
  • query_bias: optional per-head shift for query, shape [head_dim].
  • key_weight: optional per-head gain for key, shape [head_dim].
  • key_bias: optional per-head shift for key, shape [head_dim].
  • head_dim: the per-head run length; must divide the trailing extent.
  • eps: stability floor; std::nullopt selects 1e-5.

Returns: {q_normed, k_normed, v}; absent inputs come back absent.

Throws

  • ClikaRT::Error: (INVALID_ARGUMENT) when head_dim does not divide the trailing extent or an affine operand is mis-shaped.

Declared in ClikaRT/compute/ops.h, line 3585