---
title: "ClikaRT::nn::Attention"
sidebar_label: "Attention"
description: "Attention module: fixed settings, optional bound tensors, one fused call per forward. The module form of ops::attention."
---

<!-- Generated by tools/doxygen2mdx from Doxygen XML. Do not edit. -->

<span class="api-kind api-kind--class">class</span>

**Header:** `ClikaRT/nn/attention.h`  
**Inherits:** [`ClikaRT::nn::Module`](./Module.md)

[Attention](./Attention.md) module: fixed settings, optional bound tensors, one fused call per [`forward`](./Attention.md#forward). The module form of [`ops::attention`](../ops/attention.md#attention).

Inputs are head-major with the head dimension innermost: query `[B, H_q, S_q, D]`, key `[B, H_kv, S_kv, D]`, value `[B, H_kv, S_kv, Dv]` with `H_kv` dividing `H_q` (grouped-query attention broadcasts each key / value head over its query group). With `num_heads` set the inputs are the hidden-folded projection outputs instead, `[B, S, H * D]`, and the result is folded back the same way. Causal attention needs `S_q <= S_kv`.

```cpp
auto attn = ClikaRT::nn::Attention::make(
    ClikaRT::nn::AttentionOptions().is_causal(true).sliding_window(4096));
auto out = attn->forward(q, k, v);            // [B, H_q, S_q, Dv]
```

## Static member functions

### make() {#make}

<div class="api-signature">

`static` `std::shared_ptr<Attention>` **`make`**([`AttentionOptions`](./AttentionOptions.md) `options` `=` `{}`)

</div>

Defaults: every option at its default ([`AttentionOptions`](./AttentionOptions.md)).

<div class="api-declared">

Declared in `ClikaRT/nn/attention.h`, line 125

</div>

## Member functions

### \~Attention() {#Attention}

<div class="api-signature">

**`~Attention`**() `override`

</div>

<div class="api-declared">

Declared in `ClikaRT/nn/attention.h`, line 129

</div>

### set\_mask() {#set_mask}

<div class="api-signature">

`void` **`set_mask`**([`Tensor`](../Tensor.md) `mask`)

</div>

Bind an attention mask every [`forward`](./Attention.md#forward) applies (the buffer `attn_mask`): broadcastable to `[B, H_q, S_q, S_kv]`; a Bool tensor keeps where true, a float tensor adds to the logits. A mask passed to [`forward`](./Attention.md#forward) takes precedence for that call. Re-binding replaces the buffer.

<div class="api-declared">

Declared in `ClikaRT/nn/attention.h`, line 137

</div>

### set\_sinks() {#set_sinks}

<div class="api-signature">

`void` **`set_sinks`**([`Tensor`](../Tensor.md) `sinks`)

</div>

Bind the learned per-head softmax sink (the parameter `sinks`, `[H_q]`): one virtual logit per head folded into every softmax denominator, so a query can attend to nothing. Binds the declared slot when `sinks` was set at [`make`](./Attention.md#make) (the shape must match), or registers the parameter otherwise.

<div class="api-declared">

Declared in `ClikaRT/nn/attention.h`, line 145

</div>

### set\_rope() {#set_rope}

<div class="api-signature">

`void` **`set_rope`**(<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`Tensor`](../Tensor.md) `cos`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`Tensor`](../Tensor.md) `sin`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`ops::RotaryMode`](../ops/index.md#RotaryMode) `mode` `=` [`ops::RotaryMode::NeoX`](../ops/index.md#RotaryMode)<br />
)

</div>

Bind rotary tables (the buffers `rope_cos` / `rope_sin`, each `[max_positions, rotary_dim / 2]` Float32, the tables [`RotaryEmbedding::cos()`](./RotaryEmbedding.md#cos) / `sin()` or [`ops::generate_rotary_cache`](../ops/generate_rotary_cache.md#generate_rotary_cache) produce): every [`forward`](./Attention.md#forward) then rotates the queries AND the keys by position (`position_ids`, else 0, 1, 2, ...) before the dot product, so the two share one sequence length: a call with `S_q != S_kv` is refused while tables are bound. `mode` is the pair layout.

<div class="api-declared">

Declared in `ClikaRT/nn/attention.h`, line 157

</div>

### to\_impl(StreamOrDevice) {#to_impl}

<div class="api-signature">

`virtual` [`Result`](../Result.md)`<void>` **`to_impl`**([`StreamOrDevice`](../StreamOrDevice.md) `where`) `override`

</div>

Move the bound tensors to a placement ([`Device`](../Device.md) / [`Stream`](../Stream.md)); the attention is rebuilt there on the next [`forward`](./Attention.md#forward).

<div class="api-declared">

Declared in `ClikaRT/nn/attention.h`, line 164

</div>

### to\_impl(DataType) {#to_impl-2}

<div class="api-signature">

`virtual` [`Result`](../Result.md)`<void>` **`to_impl`**([`DataType`](../../index.md#DataType) `dtype`) `override`

</div>

Cast the bound `sinks` vector and a FLOAT mask to [`dtype`](../dtype/index.md); a Bool mask and the rotary tables (always Float32) keep their dtypes.

<div class="api-declared">

Declared in `ClikaRT/nn/attention.h`, line 167

</div>

### forward() {#forward}

<div class="api-signature">

[`Tensor`](../Tensor.md) **`forward`**(<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`Tensor`](../Tensor.md) `query`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`Tensor`](../Tensor.md) `key`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`Tensor`](../Tensor.md) `value`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`OptionalTensor`](../../index.md#OptionalTensor) `mask` `=` `{}`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`OptionalTensor`](../../index.md#OptionalTensor) `position_ids` `=` `{}`<br />
) `const`

</div>

[`forward_impl`](./Attention.md#forward), unwrapped: raises [`ClikaRT::Error`](../Error.md) on failure.

<div class="api-declared">

Declared in `ClikaRT/nn/attention.h`, line 186

</div>

### operator()() {#operator-call}

<div class="api-signature">

[`Tensor`](../Tensor.md) **`operator()`**(<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`Tensor`](../Tensor.md) `query`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`Tensor`](../Tensor.md) `key`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`Tensor`](../Tensor.md) `value`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`OptionalTensor`](../../index.md#OptionalTensor) `mask` `=` `{}`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`OptionalTensor`](../../index.md#OptionalTensor) `position_ids` `=` `{}`<br />
) `const`

</div>

Same as [`forward`](./Attention.md#forward).

<div class="api-declared">

Declared in `ClikaRT/nn/attention.h`, line 193

</div>

### options() {#options}

<div class="api-signature">

`const` [`AttentionOptions`](./AttentionOptions.md)`&` **`options`**() `const noexcept`

</div>

The settings this module was made with.

<div class="api-declared">

Declared in `ClikaRT/nn/attention.h`, line 201

</div>

## Protected member functions

### initialize\_impl() {#initialize_impl}

<div class="api-signature">

`virtual` [`Result`](../Result.md)`<void>` **`initialize_impl`**() `override`

</div>

The pack hook: prepares the fused attention (idempotent, thread-safe). A declared `sinks` slot must be bound first.

<div class="api-declared">

Declared in `ClikaRT/nn/attention.h`, line 211

</div>