ClikaRT::ops::attention_over_cache

function

attention_over_cache()

Tensor attention_over_cache(
    Tensor q,
    Tensor cache_key,
    Tensor cache_value,
    Tensor kvcache_start,
    Tensor cu_seqlens_q,
    Tensor cu_seqlens_k,
    ScalarOrTensor max_seqlen_q = {},
    ScalarOrTensor max_seqlen_k = {},
    OptionalTensor rope_cos = {},
    OptionalTensor rope_sin = {},
    OptionalTensor position_ids = {},
    OptionalTensor attn_mask = {},
    std::optional<bool> is_causal = std::nullopt,
    ScalarOrTensor q_scale = {},
    std::optional<double> softcap = std::nullopt,
    std::optional<int64_t> sliding_window = std::nullopt,
    std::optional<bool> smooth_softmax = std::nullopt,
    std::optional<RotaryMode> rotary_mode = std::nullopt,
    std::optional<int64_t> num_heads = std::nullopt,
    std::optional<int64_t> kv_num_heads = std::nullopt,
    ScalarOrTensor k_scale = {},
    ScalarOrTensor v_scale = {},
    OptionalTensor head_sink = {},
    OptionalTensor q_norm_gain = {},
    OptionalTensor k_norm_gain = {},
    std::optional<double> qk_norm_eps = std::nullopt,
    OptionalTensor slot_ids = {}
)

Attend a KV cache WITHOUT appending to it (a read-only re-attention). q is packed varlen [ΣS_q, H, D] (or hidden-folded [ΣS_q, num_heads*D] with num_heads set); cache_key/cache_value are a cache another attention call already appended: continuous head-major [max_seqs, H_kv, max_seq, D] with a rank-1 [B] kvcache_start (the layout selector; its values are not read, and q's position derives from cu_seqlens_k), or a paged block pool [num_blocks, H_kv, block_size, D] with a rank-2 [B, max_blocks] block table. On the continuous cache slot_ids ([B] Int32, in range, pairwise distinct) names each batch row's cache row; absent, batch row b reads cache row b. cu_seqlens_k is each sequence's TOTAL cached length: per-seq [B] or cumulative [B+1]. The cache is never written. When rope_cos/rope_sin are bound, rotary applies to q only (the cached keys are already rotated). Use case: a q-only module re-attending a sibling layer's cache. head_sink is the per-head softmax sink [H_q], a virtual logit folded into the softmax denominator, the same contract as the varlen face; it rides the parameter tail here.

Returns: out_attn: q's shape with the trailing dim set to the cache's value head size. The read-only face's qk-norm tail rotates and norms q ONLY; the cached keys were already rotated+normed by the call that appended them; k_norm_gain rides along untouched so one argument set serves both the appending and the read-only attention. Same travel rule: gains come WITH qk_norm_eps and require the rope planes; absent ⇒ unchanged.

Declared in ClikaRT/compute/ops.h, line 1305