Skip to main content

ClikaRT::ops::attention_varlen

function

attention_varlen()​

Tensor attention_varlen(
    Tensor q,
    Tensor k,
    Tensor v,
    Tensor cu_seqlens_q,
    Tensor cu_seqlens_k,
    ScalarOrTensor max_seqlen_q = {},
    ScalarOrTensor max_seqlen_k = {},
    OptionalTensor attn_mask = {},
    OptionalTensor head_sink = {},
    std::optional<bool> is_causal = std::nullopt,
    ScalarOrTensor q_scale = {},
    std::optional<double> softcap = std::nullopt,
    std::optional<int64_t> sliding_window = std::nullopt,
    std::optional<bool> smooth_softmax = std::nullopt,
    ScalarOrTensor k_scale = {},
    ScalarOrTensor v_scale = {}
)

Variable-length (packed) form of attention: the serving riders over token-packed ragged batches.

Tensors and offsets follow scaled_dot_product_attention_varlen; the riders (head_sink, softcap, sliding_window, smooth_softmax) follow attention.

Parameters

  • q: packed queries [Sum(S_q), H_q, D].
  • k: packed keys [Sum(S_kv), H_kv, D].
  • v: packed values [Sum(S_kv), H_kv, Dv].
  • cu_seqlens_q: [B+1] Int32 query prefix sums.
  • cu_seqlens_k: [B+1] Int32 KV prefix sums.
  • max_seqlen_q: optional host bound (int or HOST-resident 0-D tensor).
  • max_seqlen_k: optional host bound, same forms.
  • attn_mask: optional, broadcastable to [B, H_q, S_q, S_kv].
  • head_sink: optional [H_q] per-head softmax sink.
  • is_causal: optional causal mask; absent = not causal.
  • q_scale: softmax pre-scale.
  • softcap: optional logit soft-cap.
  • sliding_window: optional window length.
  • smooth_softmax: optional; true adds 1 to the denominator.
  • k_scale: quantized-key dequant scale.
  • v_scale: quantized-value dequant scale.

Throws

  • ClikaRT::Error: when the input's dtype/shape is not served for this operation (the machine-readable reason rides code_name()).

Returns: packed [Sum(S_q), H_q, Dv], dtype of q.

Declared in ClikaRT/compute/ops.h, line 1254