---
title: "ClikaRT::ops::attention"
sidebar_label: "attention"
description: "Dense attention with the serving riders: per-head sink, logit soft-cap, sliding window, smoothed softmax."
---

<!-- Generated by tools/doxygen2mdx from Doxygen XML. Do not edit. -->

<span class="api-kind api-kind--function">function</span>

## attention() {#attention}

<div class="api-signature">

[`Tensor`](../Tensor.md) **`attention`**(<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`Tensor`](../Tensor.md) `query`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`Tensor`](../Tensor.md) `key`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`Tensor`](../Tensor.md) `value`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`OptionalTensor`](../../index.md#OptionalTensor) `attn_mask` `=` `{}`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`OptionalTensor`](../../index.md#OptionalTensor) `head_sink` `=` `{}`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::optional<bool>` `is_causal` `=` `std::nullopt`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`ScalarOrTensor`](../ScalarOrTensor.md) `q_scale` `=` `{}`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::optional<double>` `softcap` `=` `std::nullopt`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::optional<int64_t>` `sliding_window` `=` `std::nullopt`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::optional<bool>` `smooth_softmax` `=` `std::nullopt`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`ScalarOrTensor`](../ScalarOrTensor.md) `k_scale` `=` `{}`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`ScalarOrTensor`](../ScalarOrTensor.md) `v_scale` `=` `{}`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`OptionalTensor`](../../index.md#OptionalTensor) `kept_prefix` `=` `{}`<br />
)

</div>

Dense attention with the serving riders: per-head sink, logit soft-cap, sliding window, smoothed softmax.

The core is [`scaled_dot_product_attention`](./scaled_dot_product_attention.md#scaled_dot_product_attention); each rider adjusts the softmax stage:

- `head_sink` `[H_q]`: a per-head virtual logit folded into the softmax denominator (attention that can "go nowhere").
- `softcap`: logits pass through `cap * `[`tanh`](./tanh.md#tanh)`(x / cap)` before the softmax.
- `sliding_window`: each query attends only the last N key positions.
- `smooth_softmax`: adds one to the softmax denominator. Default `false`.

**Parameters**

- `query`: queries `[B, H_q, S_q, D]`.
- `key`: keys `[B, H_kv, S_kv, D]`.
- `value`: values `[B, H_kv, S_kv, Dv]`.
- `attn_mask`: optional, broadcastable to `[B, H_q, S_q, S_kv]`.
- `head_sink`: optional `[H_q]` per-head softmax sink.
- `is_causal`: optional causal mask; absent = not causal.
- `q_scale`: softmax pre-scale (see [`scaled_dot_product_attention`](./scaled_dot_product_attention.md#scaled_dot_product_attention)).
- `softcap`: optional logit soft-cap value.
- `sliding_window`: optional window length in key positions.
- `smooth_softmax`: optional; `true` adds 1 to the softmax denominator.
- `k_scale`: quantized-key dequant scale (folds into the logits).
- `v_scale`: quantized-value dequant scale.
- `kept_prefix`: optional: each sequence's first P keys stay attended beside the sliding window (a 0-D value for every sequence, or `[B]` one per sequence; Int32 or Int64, on the host or the device). Under a causal mask a key is attended when it is at or before the query and either inside the first P keys or inside the window; with no window, or P covering the sequence, the call is plain causal. A negative P refuses. Distinct from `head_sink`, and composes with it.

**Throws**

- `ClikaRT::Error`: when the input's dtype/shape is not served for this operation (the machine-readable reason rides `code_name()`).

**Returns:** `[B, H_q, S_q, Dv]`, dtype of `query`.

<div class="api-declared">

Declared in `ClikaRT/compute/ops.h`, line 1247

</div>