---
title: "ClikaRT::ops::group_query_attention"
sidebar_label: "group_query_attention"
description: "Fused GQA with RoPE + KV cache. Bind out_present_key/out_present_value to the same buffers as past_key/past_value (a KVCache::keys/values(layer) view) + pass kvcache_start to append the new post-RoPE…"
---

<!-- Generated by tools/doxygen2mdx from Doxygen XML. Do not edit. -->

<span class="api-kind api-kind--function">function</span>

## group\_query\_attention() {#group_query_attention}

<div class="api-signature">

`std::array<`[`Tensor`](../Tensor.md) `3>` **`group_query_attention`**(<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`Tensor`](../Tensor.md) `query`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`Tensor`](../Tensor.md) `key`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`Tensor`](../Tensor.md) `value`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`OptionalTensor`](../../index.md#OptionalTensor) `past_key` `=` `{}`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`OptionalTensor`](../../index.md#OptionalTensor) `past_value` `=` `{}`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`OptionalTensor`](../../index.md#OptionalTensor) `kvcache_start` `=` `{}`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`OptionalTensor`](../../index.md#OptionalTensor) `rope_cos` `=` `{}`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`OptionalTensor`](../../index.md#OptionalTensor) `rope_sin` `=` `{}`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`OptionalTensor`](../../index.md#OptionalTensor) `position_ids` `=` `{}`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`OptionalTensor`](../../index.md#OptionalTensor) `attn_mask` `=` `{}`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::optional<bool>` `is_causal` `=` `std::nullopt`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`ScalarOrTensor`](../ScalarOrTensor.md) `q_scale` `=` `{}`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::optional<double>` `softcap` `=` `std::nullopt`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::optional<int64_t>` `sliding_window` `=` `std::nullopt`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::optional<bool>` `smooth_softmax` `=` `std::nullopt`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::optional<`[`RotaryMode`](./index.md#RotaryMode)`>` `rotary_mode` `=` `std::nullopt`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::optional<int64_t>` `num_heads` `=` `std::nullopt`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::optional<int64_t>` `kv_num_heads` `=` `std::nullopt`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`OptionalTensor`](../../index.md#OptionalTensor) `out_present_key` `=` `{}`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`OptionalTensor`](../../index.md#OptionalTensor) `out_present_value` `=` `{}`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`ScalarOrTensor`](../ScalarOrTensor.md) `k_scale` `=` `{}`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`ScalarOrTensor`](../ScalarOrTensor.md) `v_scale` `=` `{}`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`OptionalTensor`](../../index.md#OptionalTensor) `head_sink` `=` `{}`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`OptionalTensor`](../../index.md#OptionalTensor) `q_norm_gain` `=` `{}`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`OptionalTensor`](../../index.md#OptionalTensor) `k_norm_gain` `=` `{}`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::optional<double>` `qk_norm_eps` `=` `std::nullopt`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`OptionalTensor`](../../index.md#OptionalTensor) `slot_ids` `=` `{}`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`OptionalTensor`](../../index.md#OptionalTensor) `kept_prefix` `=` `{}`<br />
)

</div>

Fused GQA with RoPE + KV cache. Bind `out_present_key`/`out_present_value` to the same buffers as `past_key`/`past_value` (a `KVCache::keys`/`values(layer)` view) + pass `kvcache_start` to append the new post-RoPE K/V IN PLACE into the cache (the decode perf path). q/k/v are hidden-folded `[ΣS, heads*head_dim]`; `num_heads`/`kv_num_heads` drive the in-op head split (GQA). `head_sink` is the per-head softmax sink `[H_q]`, a virtual logit folded into the softmax denominator (attention-sink models bind one per layer), the same contract as [`group_query_attention_varlen`](./group_query_attention_varlen.md#group_query_attention_varlen); it rides the parameter tail here. `q_norm_gain`/`k_norm_gain` engage the POST-rope per-head RMS norm: after the in-op rotation, every head's `[head_dim]` q (and new-k) vector is RMS-normalized with the gain BEFORE any cache append, so the cache holds rotated+normed keys. Each gain is rank-1 `[head_dim]` (one vector shared across heads; any other shape rejects), any float dtype, applied at its own dtype. The gains require the in-op rope planes (`rope_cos`/`rope_sin`), and they travel WITH `qk_norm_eps`: pass the model's own rms-norm epsilon alongside the gains, or neither (a gain without the epsilon, or an epsilon with no gain, rejects). Absent ⇒ the gain-less path, unchanged. A rope-free per-head norm composes [`qk_rms_norm`](./qk_rms_norm.md#qk_rms_norm) instead. `slot_ids` (`[B]` Int32) names the cache row each batch row appends to and attends from on a continuous `[max_seqs, H_kv, max_seq, D]` cache: a sequence keeps its row while the batch composition changes around it. Absent, batch row b uses cache row b. Every entry must lie in `[0, max_seqs)` and no two rows may share one (each rejects). On that cache `kvcache_start` stays the rank-1 `[B]` layout selector whose values are not read: each row's write offset is `cu_seqlens_k[b] - q_len[b]`. A paged block table and the dense in-place form take no `slot_ids` (the block table is its own row map; the dense form addresses rows by batch index). A paged block table (`[B, max_blocks]` Int32) is validated whole before any read: every entry is `-1` (the unused-tail pad) or a block index in `[0, num_blocks)`, and no two entries of one row name the same physical block; an out-of-range entry and a repeated one alike refuse `INVALID_ARGUMENT` (a repeat would alias two positions onto one block and decode wrong values silently). A read-only attend (`key` and `value` absent) takes no present outputs: use [`attention_over_cache`](./attention_over_cache.md#attention_over_cache), or leave `out_present_key`/`out_present_value` unbound, and `present_key`/`present_value` come back undefined. A bound present refuses `INVALID_ARGUMENT`. `kept_prefix` keeps each sequence's first P keys attended beside the sliding window (a 0-D value for every sequence, or `[B]` one per sequence; Int32 or Int64): under the causal bound a key is attended when it is inside the first P keys or inside the window; absent, the window alone. A negative P refuses.

**Returns:** \{out\_attn, present\_key, present\_value\}.

<div class="api-declared">

Declared in `ClikaRT/compute/ops.h`, line 1322

</div>