---
title: "ClikaRT::nn::KVCache"
sidebar_label: "KVCache"
description: "The serving-side KV cache: per-layer storage rows (attention K/V, conv windows, recurrent slabs) over a slot-indexed batch."
---

<!-- Generated by tools/doxygen2mdx from Doxygen XML. Do not edit. -->

<span class="api-kind api-kind--class">class</span>

**Header:** `ClikaRT/nn/kv_cache.h`

The serving-side KV cache: per-layer storage rows (attention K/V, conv windows, recurrent slabs) over a slot-indexed batch.

Built once per serving configuration with `make(config)`; the attention ops append each step's K/V through it and attend over what it holds. Rows lay out CONTINUOUS (per-slot buffers) or PAGED (a shared block pool) per the config's mode; a [`KVQuantSpec`](./KVQuantSpec.md) on a row quantizes on append and dequantizes on attend, invisibly to the caller. Move-only handle; the cache owns its device storage for its whole lifetime.

```cpp
auto cache = ClikaRT::nn::KVCache::make(config);
// per step: the attention ops read/write through the cache's tensors
```

## Nested types

| Name | Description |
| --- | --- |
| [`StepIndices`](./KVCache.StepIndices.md) | The device-side index tensors one varlen decode/prefill step consumes, derived from the host-side step description (see [`prepare_step`](./KVCache.md#prepare_step)). |

## Static member functions

### make() {#make}

<div class="api-signature">

`static` [`KVCache`](./KVCache.md) **`make`**(`const` [`KVCacheConfig`](./KVCacheConfig.md)`&` `config`, [`Span`](../Span.md)`<const` [`KVLayerSpec`](./KVLayerSpec.md)`>` `layer_specs`)

</div>

Build a cache with one layer per `layer_specs` row (size must equal `config.num_layers`). Every composition goes through this one entry: a uniform full-attention stack is N identical `AttentionKV` rows; a windowed/full alternation varies `window` per row; recurrent stacks mix in the state kinds. Each row's `Layout::Auto` resolves from `config.mode`; a row may pin `Continuous`/`Paged` explicitly and mix freely with the rest. When any row lays out `Paged`, `config.paged` must carry the pool geometry; paged rows share one block table, with one pool per distinct (kv\_heads, head\_dim) row geometry. Container bookkeeping ([`seq_lens`](./KVCache.md#seq_lens)) reads a paged row when one exists, else the first token-indexed layer, so a stack whose layer 0 is a state row works.

Raises [`ClikaRT::Error`](../Error.md) on a spec the cache cannot build: a spec/num\_layers size mismatch, an empty state geometry, `max_tokens_per_seq <= 0` with a non-windowed continuous row present, a paged row asking for an asymmetric `v_head_dim` (it stays on continuous rows), a negative `kv_heads`, a per-row `k_quant`/`v_quant` override, or a per-row `k_scale`/`v_scale` calibration override that breaks its contract (see the field).

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 486

</div>

### footprint() {#footprint}

<div class="api-signature">

`static` [`KVCacheFootprint`](./KVCacheFootprint.md) **`footprint`**(`const` [`KVCacheConfig`](./KVCacheConfig.md)`&` `config`, [`Span`](../Span.md)`<const` [`KVLayerSpec`](./KVLayerSpec.md)`>` `layer_specs`)

</div>

What `make(config, layer_specs)` would allocate, by storage class (see [`KVCacheFootprint`](./KVCacheFootprint.md)), without allocating: the composition resolves exactly as [`make`](./KVCache.md#make) resolves it and every figure is read off the plan the cache builds from. Raises [`ClikaRT::Error`](../Error.md) on every spec [`make`](./KVCache.md#make) raises on, with the same message, so a sizing pass refuses a composition before any byte moves.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 497

</div>

## Member functions

### KVCache() {#KVCache}

<div class="api-signature">

**`KVCache`**(`KVCache&&` `other`) `noexcept`

</div>

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 500

</div>

### operator=() {#operator-assign}

<div class="api-signature">

[`KVCache`](./KVCache.md)`&` **`operator=`**([`KVCache`](./KVCache.md)`&&` `other`) `noexcept`

</div>

Move-only handle; the moved-from object is empty.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 502

</div>

### \~KVCache() {#KVCache-2}

<div class="api-signature">

**`~KVCache`**()

</div>

Releases the cache's device storage.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 504

</div>

### keys() {#keys}

<div class="api-signature">

[`Tensor`](../Tensor.md) **`keys`**(`int` `layer`) `const`

</div>

This layer's K / V storage view; bind as `past_key`/`out_present_key` (and value) into `group_query_attention_varlen` (a read-only attend binds `past_key`/`past_value` only). Raises [`ClikaRT::Error`](../Error.md) (`InvalidArgument`) when `layer` is outside `[0, `[`num_layers`](./KVCache.md#num_layers)`)`; the message names the index and the layer count.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 512

</div>

### values() {#values}

<div class="api-signature">

[`Tensor`](../Tensor.md) **`values`**(`int` `layer`) `const`

</div>

The value-side sibling of `keys(layer)`: this layer's V storage view, same binding use.

**Throws**

- `ClikaRT::Error`: as [`keys`](./KVCache.md#keys) does.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 516

</div>

### conv\_state() {#conv_state}

<div class="api-signature">

[`Tensor`](../Tensor.md) **`conv_state`**(`int` `layer`, `int` `index` `=` `0`) `const`

</div>

A State layer's i-th conv window slab `[max_seqs, dim_i, width_i]`, a shared handle; bind it (or a slot-row view of it) as `causal_conv_update`'s state and the op updates it in place. Raises on an out-of-range `layer`, a non-State layer, or an out-of-range `index`.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 524

</div>

### recurrent\_state() {#recurrent_state}

<div class="api-signature">

[`Tensor`](../Tensor.md) **`recurrent_state`**(`int` `layer`) `const`

</div>

A State layer's recurrent slab `[max_seqs, heads, k_dim, v_dim]`. Raises on an out-of-range `layer`, a non-State layer, or one that declared no recurrent state.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 530

</div>

### seq\_lens() {#seq_lens}

<div class="api-signature">

[`Tensor`](../Tensor.md) **`seq_lens`**() `const`

</div>

Per-slot effective token counts, `[max_seqs]` Int32 on the cache device.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 534

</div>

### mode() {#mode}

<div class="api-signature">

[`KVCacheMode`](./index.md#KVCacheMode) **`mode`**() `const`

</div>

The serving mode this cache was built with, the value every `Layout::Auto` row resolved to. The predicate that selects the decode addressing path (cheap, allocates nothing).

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 539

</div>

### num\_layers() {#num_layers}

<div class="api-signature">

`int` **`num_layers`**() `const`

</div>

The number of layers this cache was built with: the valid `layer` range of [`keys`](./KVCache.md#keys) / [`values`](./KVCache.md#values) / [`conv_state`](./KVCache.md#conv_state) / [`recurrent_state`](./KVCache.md#recurrent_state) is `[0, `[`num_layers()`](./KVCache.md#num_layers)`)`. Never fails; allocates nothing.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 543

</div>

### paged\_block\_table() {#paged_block_table}

<div class="api-signature">

[`Tensor`](../Tensor.md) **`paged_block_table`**() `const`

</div>

The paged block table, `[max_seqs, max_blocks_per_seq]` Int32, or an undefined tensor for a cache with no paged rows. Use [`mode()`](./KVCache.md#mode) for the serving-mode test; this accessor is for gathering a sequence's blocks.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 548

</div>

### paged\_free\_blocks() {#paged_free_blocks}

<div class="api-signature">

`std::int64_t` **`paged_free_blocks`**() `const`

</div>

Blocks of the shared paged pool no live slot holds right now, or -1 for a cache with no paged rows. A parked block (a cached prefix no slot is bound to) counts as free: the next [`reserve`](./KVCache.md#reserve) or [`admit`](./KVCache.md#admit) that needs room takes it, least recently parked first. A host-side counter read; it allocates nothing and waits on nothing, so a scheduler may read it once per admission round ([`runtime::Model::capacity_units`](../runtime/Model.md#capacity_units)).

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 556

</div>

### paged\_num\_blocks() {#paged_num_blocks}

<div class="api-signature">

`std::int64_t` **`paged_num_blocks`**() `const`

</div>

The shared paged pool's size in blocks ([`PagedParams::num_blocks`](./PagedParams.md#num_blocks)), or -1 for a cache with no paged rows. Never fails; allocates nothing.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 559

</div>

### reserve() {#reserve}

<div class="api-signature">

[`Tensor`](../Tensor.md) **`reserve`**([`Tensor`](../Tensor.md) `slot_ids`, [`Span`](../Span.md)`<const std::int32_t>` `n_new_per_seq`)

</div>

Paged only: reserve `n_new_per_seq[i]` new tokens for slot `slot_ids[i]`, ONCE per decode step (acquire blocks + advance seq\_len). Returns the per-sequence pre-advance write offset, `[B]` Int32 on the cache device. Raises [`ClikaRT::Error`](../Error.md) on a continuous cache or pool exhaustion.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 566

</div>

### prepare\_step() {#prepare_step}

<div class="api-signature">

[`StepIndices`](./KVCache.StepIndices.md) **`prepare_step`**(<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`Span`](../Span.md)`<const std::int32_t>` `cu_seqlens_q`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`Span`](../Span.md)`<const std::int32_t>` `past_len`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`Span`](../Span.md)`<const std::int32_t>` `slot_ids`<br />
)

</div>

Convert one step's host-side description (the `[B+1]` query cumulative `cu_seqlens_q`, the `[B]` per-slot committed lengths `past_len`, and the `[B]` cache slot ids) into the device-side index tensors a varlen attention forward binds. One call per step, before the layer loop:

- On a PAGED cache this also RESERVES the step's blocks (once, covering every layer; do not call [`reserve`](./KVCache.md#reserve) separately) and returns the block table gathered into slot order as `kvcache_start` (`[B, blocks_per_seq]`); `cu_seqlens_k` is `[B]` post-append lengths in decode (every query length 1) or the `[B+1]` cumulative in prefill.
- On a CONTINUOUS cache `slot_ids` is the `[B]` slot ids on device (the attention op's `slot_ids` input: the cache row each batch row appends to and attends from), `cu_seqlens_k` is the `[B]` per-seq post-append lengths, and `kvcache_start` is the rank-1 `[B]` selector the op does not read the values of (it carries the same slot ids; every write offset derives from `cu_seqlens_k`). Bind all three to the attention call; on a paged cache `slot_ids` is undefined and binds as absent.

Steady-state decode repeats the same `cu_seqlens_q` / slot list every step; those uploads are cached by content and re-run only when the batch composition changes. Drive a cache's steps from ONE thread (the uploads are cached without locking). Raises [`ClikaRT::Error`](../Error.md) on mismatched span lengths or pool exhaustion.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 639

</div>

### admit() {#admit}

<div class="api-signature">

`std::int64_t` **`admit`**(<br />
&nbsp;&nbsp;&nbsp;&nbsp;`int` `batch_idx`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`Tensor`](../Tensor.md) `prompt_ids`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::int64_t` `addressable_len` `=` [`kAddressableAll`](./KVCache.md#kAddressableAll)<br />
)

</div>

Paged only: admit a freshly-recycled slot with its prompt. Hashes the prompt's full blocks, binds the longest already-cached prefix into the slot (sharing physical blocks with the sessions that produced them), and returns the number of leading tokens whose K/V is already cached, always block-aligned and strictly less than the prompt length. Forward only `prompt_ids[num_cached:]` with past length `num_cached`. Call once per session, before the first [`reserve`](./KVCache.md#reserve). `prompt_ids` is `[N]` Int32 (any device). Raises [`ClikaRT::Error`](../Error.md) on a continuous cache or a non-fresh slot.

`addressable_len` (default [`kAddressableAll`](./KVCache.md#kAddressableAll)): the count of leading tokens whose K/V depends on the token ids alone. A multimodal prompt passes the position of its FIRST media token; caching then covers only the pure-text prefix before it (blocks at or past the boundary are never shared), at zero cost to text-only sessions. Pass the SAME value to this session's [`retire`](./KVCache.md#retire).

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 669

</div>

### retire() {#retire}

<div class="api-signature">

`void` **`retire`**(<br />
&nbsp;&nbsp;&nbsp;&nbsp;`int` `batch_idx`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`Tensor`](../Tensor.md) `full_token_ids`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::int64_t` `addressable_len` `=` [`kAddressableAll`](./KVCache.md#kAddressableAll),<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`Span`](../Span.md)`<const std::int64_t>` `boundaries` `=` `{}`<br />
)

</div>

Paged only: retire a FINISHED session gracefully. Pass the session's full committed token history (`[N]` Int32, prompt + generated, exactly the tokens whose K/V the slot holds): their blocks stay cached for future [`admit`](./KVCache.md#admit)s (a follow-up chat turn resubmits the transcript, so its prefix hits), then the slot is freed. Use [`evict_batch`](./KVCache.md#evict_batch) instead on the cancel/error path: it caches nothing new (the generated tokens' blocks return to the pool) and parks the prompt blocks this session's [`admit`](./KVCache.md#admit) already cached; [`reset`](./KVCache.md#reset) is the call that drops those too. `addressable_len` as on [`admit`](./KVCache.md#admit): blocks at or past the boundary are dropped, never cached.

`boundaries` (default none): further lengths inside the history at which a later prompt may stop and still be served up to that length, when the cache would otherwise serve only the whole blocks before it: a multimodal session names the length of the text before its first media item and the end of each item's rows, so a later prompt with the same text and another picture is served that text, and one with the same picture followed by a short question is served the picture. A length on a block edge or past the history adds nothing. The cache keeps at most four such keys per block (counted with the trailing partial block's own, under one parent block): a history naming more inside one block keeps the latest named, and the earliest leave.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 697

</div>

### fill() {#fill}

<div class="api-signature">

`void` **`fill`**(<br />
&nbsp;&nbsp;&nbsp;&nbsp;`int` `layer`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`int` `batch_idx`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`Tensor`](../Tensor.md) `k`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`Tensor`](../Tensor.md) `v`<br />
)

</div>

Fill layer `layer`'s K/V planes for slot `batch_idx` in ONE call, the write verb for `Kind::CrossAttentionKV` rows (encoder cross-attention: project the encoder states per layer, hand both projections here). `k` / `v` are token-major `[t_enc, kv_heads, head_dim]`; a dtype differing from the cache's kv\_dtype converts inside the copy. Sets the filled layer's slot length to `t_enc`, IDEMPOTENTLY: a refill with the SAME `t_enc` overwrites in place; a different `t_enc` on a non-empty slot raises (evict the slot first). Raises on a non-CrossAttentionKV layer, a shape/rank mismatch, or `t_enc` beyond `max_tokens_per_seq`.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 713

</div>

### update\_position() {#update_position}

<div class="api-signature">

`void` **`update_position`**(`int` `batch_idx`, `std::int64_t` `new_pos`)

</div>

Set the effective seq\_len of `batch_idx` to `new_pos` (bookkeeping only; does not trim storage). Call after each step's append. A rewind below the position a step reserved (a speculative step's rejected tokens) keeps a paged row's blocks bound to the slot: [`paged_free_blocks()`](./KVCache.md#paged_free_blocks) does not move, the next [`prepare_step`](./KVCache.md#prepare_step) re-serves those blocks before acquiring new ones, the rewound positions take the next append and read back after it, and [`evict_batch`](./KVCache.md#evict_batch) returns every block. A windowed paged row bounds the rewind's depth instead (`PagedParams::window`). Raises on error.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 725

</div>

### evict\_batch() {#evict_batch}

<div class="api-signature">

`void` **`evict_batch`**(`int` `batch_idx`)

</div>

Free a slot's stored K/V (its [`seq_lens()`](./KVCache.md#seq_lens) entry returns to 0). On a paged cache with prefix caching this is the cancel/error path: the slot's blocks that no cache hash names return to the pool, and its sealed prefix blocks (the full prompt blocks an [`admit`](./KVCache.md#admit) enrolled) are parked for a later [`admit`](./KVCache.md#admit) of the same prompt; nothing new is cached. [`reset`](./KVCache.md#reset) is the call that drops the parked blocks too.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 734

</div>

### abandon\_batch() {#abandon_batch}

<div class="api-signature">

`void` **`abandon_batch`**(`int` `batch_idx`)

</div>

Free a slot after a FAILED session, with host-side bookkeeping only: no kernel launch, no device copy, no stream wait, whatever the cache carries. Block references the host bookkeeping knows are released exactly as [`evict_batch`](./KVCache.md#evict_batch) releases them (a shared prefix block other slots reference is untouched); anything newer is dropped, bounded to this slot, and never listed as free again. The slot's storage hygiene runs at its next use, on that caller's stream, before any read. Never fails for device reasons; a bad slot index reports INVALID\_ARGUMENT. Prefer [`evict_batch`](./KVCache.md#evict_batch) when the session ended normally (it keeps the pool's bookkeeping exact); use this when the session ended in an error its device may not have survived.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 748

</div>

### reset() {#reset}

<div class="api-signature">

`void` **`reset`**()

</div>

Drop all stored K/V across every slot and layer, the parked prefix blocks included: after [`reset`](./KVCache.md#reset) the cache holds no cached state, and the next [`admit`](./KVCache.md#admit) of any prompt reports zero cached tokens.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 754

</div>

## Static data members

### kAddressableAll {#kAddressableAll}

<div class="api-signature">

`static` `std::int64_t` **`kAddressableAll`** `= -1`

</div>

The [`admit`](./KVCache.md#admit)/[`retire`](./KVCache.md#retire) addressability default: every token's K/V is a pure function of the token ids, so the whole prompt participates in prefix caching, the text-only posture. A MULTIMODAL session must pass its first media position instead (see [`admit`](./KVCache.md#admit)): the K/V under a media placeholder comes from pixels/audio the ids cannot identify, so id-keyed sharing at or past it would serve another session's media content.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 650

</div>