ClikaRT::ops::attention

function

attention()

Tensor attention(
    Tensor q,
    Tensor k,
    Tensor v,
    OptionalTensor attn_mask = {},
    OptionalTensor head_sink = {},
    std::optional<bool> is_causal = std::nullopt,
    ScalarOrTensor q_scale = {},
    std::optional<double> softcap = std::nullopt,
    std::optional<int64_t> sliding_window = std::nullopt,
    std::optional<bool> smooth_softmax = std::nullopt,
    ScalarOrTensor k_scale = {},
    ScalarOrTensor v_scale = {}
)

Dense attention with the serving riders: per-head sink, logit soft-cap, sliding window, smoothed softmax.

The core is scaled_dot_product_attention; each rider adjusts the softmax stage:

head_sink [H_q]: a per-head virtual logit folded into the softmax denominator (attention that can "go nowhere").
softcap: logits pass through cap * tanh(x / cap) before the softmax.
sliding_window: each query attends only the last N key positions.
smooth_softmax: adds one to the softmax denominator. Default false.

Parameters

q: queries [B, H_q, S_q, D].
k: keys [B, H_kv, S_kv, D].
v: values [B, H_kv, S_kv, Dv].
attn_mask: optional, broadcastable to [B, H_q, S_q, S_kv].
head_sink: optional [H_q] per-head softmax sink.
is_causal: optional causal mask; absent = not causal.
q_scale: softmax pre-scale (see scaled_dot_product_attention).
softcap: optional logit soft-cap value.
sliding_window: optional window length in key positions.
smooth_softmax: optional; true adds 1 to the softmax denominator.
k_scale: quantized-key dequant scale (folds into the logits).
v_scale: quantized-value dequant scale.

Throws

ClikaRT::Error: when the input's dtype/shape is not served for this operation (the machine-readable reason rides code_name()).

Returns: [B, H_q, S_q, Dv], dtype of q.

Declared in ClikaRT/compute/ops.h, line 1227

attention()​

attention()