---
title: "ClikaRT::nn::KVCacheConfig"
sidebar_label: "KVCacheConfig"
description: "Construction parameters shared by every cache strategy. num_kv_heads is the KV head count (grouped-query attention: KV heads ≤ query heads); the cache stores K/V as [seq_len, num_kv_heads, head_dim] p…"
---

<!-- Generated by tools/doxygen2mdx from Doxygen XML. Do not edit. -->

<span class="api-kind api-kind--struct">struct</span>

**Header:** `ClikaRT/nn/kv_cache.h`

Construction parameters shared by every cache strategy. [`num_kv_heads`](./KVCacheConfig.md#num_kv_heads) is the KV head count (grouped-query attention: KV heads ≤ query heads); the cache stores K/V as `[seq_len, `[`num_kv_heads`](./KVCacheConfig.md#num_kv_heads)`, `[`head_dim`](./KVCacheConfig.md#head_dim)`]` per (slot, layer).

## Data members

### num\_layers {#num_layers}

<div class="api-signature">

`int32_t` **`num_layers`** `= 0`

</div>

Decoder layer count (one spec row per layer).

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 341

</div>

### num\_kv\_heads {#num_kv_heads}

<div class="api-signature">

`int32_t` **`num_kv_heads`** `= 0`

</div>

Default KV head count for rows that leave [`KVLayerSpec::kv_heads`](./KVLayerSpec.md#kv_heads) at 0.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 344

</div>

### head\_dim {#head_dim}

<div class="api-signature">

`int32_t` **`head_dim`** `= 0`

</div>

Default per-head width for rows that leave their head\_dim fields at 0.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 347

</div>

### kv\_dtype {#kv_dtype}

<div class="api-signature">

[`DataType`](../../index.md#DataType) **`kv_dtype`** `=` [`DataType::BFloat16`](../../index.md#DataType)

</div>

Storage dtype for unquantized K/V planes.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 349

</div>

### max\_seqs {#max_seqs}

<div class="api-signature">

`int32_t` **`max_seqs`** `= 1`

</div>

concurrent batch slots

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 350

</div>

### max\_tokens\_per\_seq {#max_tokens_per_seq}

<div class="api-signature">

`int64_t` **`max_tokens_per_seq`** `= 0`

</div>

per-slot token cap (the preallocated depth)

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 351

</div>

### device {#device}

<div class="api-signature">

[`Device`](../Device.md) **`device`** `=` [`Device::cpu`](../Device.md#cpu)`()`

</div>

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 352

</div>

### k\_quant {#k_quant}

<div class="api-signature">

`std::optional<`[`KVQuantSpec`](./KVQuantSpec.md)`>` **`k_quant`**

</div>

quantize the K side (see [KVQuantSpec](./KVQuantSpec.md))

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 353

</div>

### v\_quant {#v_quant}

<div class="api-signature">

`std::optional<`[`KVQuantSpec`](./KVQuantSpec.md)`>` **`v_quant`**

</div>

quantize the V side

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 354

</div>

### kv\_cache\_scheme {#kv_cache_scheme}

<div class="api-signature">

`std::string` **`kv_cache_scheme`**

</div>

Checkpoint-convention KV quantization: the fields a quantized checkpoint's config declares, bound here so a loader passes them through instead of re-deriving typed specs:

- [`kv_cache_scheme`](./KVCacheConfig.md#kv_cache_scheme) names a BLOCK storage scheme for BOTH sides by its scheme string (e.g. "GGUF\_Q8\_0", "NVFP4\_E4M3"); the string form reaches the runtime's full block-scheme taxonomy without a wide enum. It is an alternative spelling of [`KVQuantSpec::block`](./KVQuantSpec.md#block); setting both on one side raises. Empty (the default) = unused. An unknown name raises. Element-coded checkpoint schemes (e.g. a per-tensor fp8 KV cache) are expressed through the typed [`KVQuantSpec`](./KVQuantSpec.md) fields instead.
- [`k_scale`](./KVCacheConfig.md#k_scale) / [`v_scale`](./KVCacheConfig.md#v_scale) are the checkpoint's per-tensor calibration scales; each feeds that side's [`KVQuantSpec::scale`](./KVQuantSpec.md#scale). The side must be element-coded quantized: a scale on an unquantized side, a side whose spec already carries a non-default `scale`, a per-token side (it derives its own row scales), or a block side (its scales ride inline; a checkpoint global there is encode-time provenance an online-encoding cache does not consume) raises rather than being dropped.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 375

</div>

### k\_scale {#k_scale}

<div class="api-signature">

`std::optional<double>` **`k_scale`**

</div>

Model-level K-side calibration scale (pairs with [`kv_cache_scheme`](./KVCacheConfig.md#kv_cache_scheme)); absent = uncalibrated.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 378

</div>

### v\_scale {#v_scale}

<div class="api-signature">

`std::optional<double>` **`v_scale`**

</div>

Model-level V-side calibration scale, [`k_scale`](./KVCacheConfig.md#k_scale)'s twin.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 380

</div>

### mode {#mode}

<div class="api-signature">

[`KVCacheMode`](./index.md#KVCacheMode) **`mode`** `=` [`KVCacheMode::Continuous`](./index.md#KVCacheMode)

</div>

The serving mode; it resolves every row's `Layout::Auto`. A composition with no token-indexed rows serves identically under both modes.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 385

</div>

### paged {#paged}

<div class="api-signature">

`std::optional<`[`PagedParams`](./PagedParams.md)`>` **`paged`**

</div>

Shared block-pool geometry. Required when any row lays out `Paged` (via [`mode`](./KVCacheConfig.md#mode) or an explicit `Layout::Paged` pin); ignored when none does. Per-layer K/V geometry (a row's own [`head_dim`](./KVCacheConfig.md#head_dim) / `v_head_dim`) rides [`KVLayerSpec`](./KVLayerSpec.md), not this config.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 390

</div>