Skip to main content

ClikaRT::ops::attention_varlen

function

attention_varlen()​

Tensor attention_varlen(
    Tensor query,
    Tensor key,
    Tensor value,
    Tensor cu_seqlens_q,
    Tensor cu_seqlens_k,
    ScalarOrTensor max_seqlen_q = {},
    ScalarOrTensor max_seqlen_k = {},
    OptionalTensor attn_mask = {},
    OptionalTensor head_sink = {},
    std::optional<bool> is_causal = std::nullopt,
    ScalarOrTensor q_scale = {},
    std::optional<double> softcap = std::nullopt,
    std::optional<int64_t> sliding_window = std::nullopt,
    std::optional<bool> smooth_softmax = std::nullopt,
    ScalarOrTensor k_scale = {},
    ScalarOrTensor v_scale = {},
    OptionalTensor kept_prefix = {},
    OptionalTensor kv_position_offset = {}
)

Variable-length (packed) form of attention: the serving riders over token-packed ragged batches.

Tensors and offsets follow scaled_dot_product_attention_varlen; the riders (head_sink, softcap, sliding_window, smooth_softmax) follow attention.

Parameters

  • query: packed queries [Sum(S_q), H_q, D].
  • key: packed keys [Sum(S_kv), H_kv, D].
  • value: packed values [Sum(S_kv), H_kv, Dv].
  • cu_seqlens_q: [B+1] Int32 query prefix sums.
  • cu_seqlens_k: [B+1] Int32 KV prefix sums.
  • max_seqlen_q: optional host bound (int or HOST-resident 0-D tensor).
  • max_seqlen_k: optional host bound, same forms.
  • attn_mask: optional, broadcastable to [B, H_q, S_q, S_kv].
  • head_sink: optional [H_q] per-head softmax sink.
  • is_causal: optional causal mask; absent = not causal.
  • q_scale: softmax pre-scale.
  • softcap: optional logit soft-cap.
  • sliding_window: optional window length.
  • smooth_softmax: optional; true adds 1 to the denominator.
  • k_scale: quantized-key dequant scale.
  • v_scale: quantized-value dequant scale.
  • kept_prefix: optional: each request's first P keys stay attended beside the sliding window (a 0-D value for every request, or [B] one per request; Int32 or Int64), the same law as attention's.
  • kv_position_offset: optional [B] (Int32 or Int64): each paged row's served-key offset. When a windowed paged cache serves only a row's recent blocks, its served key at view position i sits at absolute position i + offset, so rotary and the window read true positions. Pass KVCache::StepIndices::kv_position_offset through as-is; it is undefined on every other cache.

Throws

  • ClikaRT::Error: when the input's dtype/shape is not served for this operation (the machine-readable reason rides code_name()).

Returns: packed [Sum(S_q), H_q, Dv], dtype of query.

Declared in ClikaRT/compute/ops.h, line 1285