ClikaRT::ops::group_query_attention_varlen

function

group_query_attention_varlen()

std::array<Tensor 3> group_query_attention_varlen(
    Tensor query,
    Tensor key,
    Tensor value,
    Tensor cu_seqlens_q,
    Tensor cu_seqlens_k,
    ScalarOrTensor max_seqlen_q = {},
    ScalarOrTensor max_seqlen_k = {},
    OptionalTensor past_key = {},
    OptionalTensor past_value = {},
    OptionalTensor kvcache_start = {},
    OptionalTensor rope_cos = {},
    OptionalTensor rope_sin = {},
    OptionalTensor position_ids = {},
    OptionalTensor attn_mask = {},
    OptionalTensor head_sink = {},
    std::optional<bool> is_causal = std::nullopt,
    ScalarOrTensor q_scale = {},
    std::optional<double> softcap = std::nullopt,
    std::optional<int64_t> sliding_window = std::nullopt,
    std::optional<bool> smooth_softmax = std::nullopt,
    std::optional<RotaryMode> rotary_mode = std::nullopt,
    std::optional<int64_t> num_heads = std::nullopt,
    std::optional<int64_t> kv_num_heads = std::nullopt,
    OptionalTensor out_present_key = {},
    OptionalTensor out_present_value = {},
    ScalarOrTensor k_scale = {},
    ScalarOrTensor v_scale = {},
    OptionalTensor q_norm_gain = {},
    OptionalTensor k_norm_gain = {},
    std::optional<double> qk_norm_eps = std::nullopt,
    OptionalTensor slot_ids = {},
    OptionalTensor kept_prefix = {},
    OptionalTensor kv_position_offset = {}
)

The varlen variant of the fused GQA above, with the same qk-norm tail: q_norm_gain/k_norm_gain (rank-1 [head_dim], post-rope, applied before the K/V append) travel with qk_norm_eps (the model's rms-norm epsilon) and require the in-op rope planes; absent ⇒ unchanged. The gains are the after-rotation order; a model that norms its heads before the rotation (the Qwen3 order) runs qk_rms_norm on its projections ahead of this op and passes no gains. The same slot_ids contract: on a continuous cache it names each batch row's cache row ([B] Int32, in range, pairwise distinct; absent = row b for batch row b), and kvcache_start's values stay unread there (the write offsets derive from cu_seqlens_k). The same block-table contract on a paged pool: every entry of the [B, max_blocks] table is -1 or a block index in [0, num_blocks), and no two entries of one row name the same physical block; an out-of-range or repeated entry refuses INVALID_ARGUMENT before any read or append. The same read-only rule: with key and value absent it takes no present outputs; use attention_over_cache, or leave out_present_key/out_present_value unbound. kept_prefix keeps each request's first P keys attended beside the sliding window (a 0-D value or [B], Int32 or Int64), the same law as attention's. kv_position_offset ([B], Int32 or Int64) is each paged row's served-key offset: when a windowed paged cache serves only a row's recent blocks, its served key at view position i sits at absolute position i + offset, so the in-op rotary and the window read true positions; pass KVCache::StepIndices::kv_position_offset through as-is (undefined on every other cache).

Declared in ClikaRT/compute/ops.h, line 1343