---
title: "ClikaRT::nn::KVQuantSpec"
sidebar_label: "KVQuantSpec"
description: "Per-side KV quantization scheme (quantize-on-append). Three families: ELEMENT-CODED (block == KVBlockScheme::None, the default): the side stores CODE bytes at dtype (Int8 / UInt8 / Float8_E4M3 / Float…"
---

<!-- Generated by tools/doxygen2mdx from Doxygen XML. Do not edit. -->

<span class="api-kind api-kind--struct">struct</span>

**Header:** `ClikaRT/nn/kv_cache.h`

Per-side KV quantization scheme (quantize-on-append). Three families:

- ELEMENT-CODED ([`block`](./KVQuantSpec.md#block)` == `[`KVBlockScheme::None`](./index.md#KVBlockScheme), the default): the side stores CODE bytes at [`dtype`](../dtype/index.md) (Int8 / UInt8 / Float8\_E4M3 / Float8\_E5M2) under the per-tensor [`scale`](./KVQuantSpec.md#scale) (+ optional integer [`zero_point`](./KVQuantSpec.md#zero_point); fp8 codes are scale-only; one there is rejected at construction).
- BLOCK-QUANTIZED ([`block`](./KVQuantSpec.md#block) set): the side stores packed blocks with inline scales (see [KVBlockScheme](./index.md#KVBlockScheme)); [`dtype`](../dtype/index.md) / [`scale`](./KVQuantSpec.md#scale) / [`zero_point`](./KVQuantSpec.md#zero_point) must stay at their defaults.
- PER-TOKEN ([`per_token`](./KVQuantSpec.md#per_token) true): the online scheme; every appended row encodes with its own absmax-derived scale, stored in a scale plane the cache allocates beside its buffer. Element-coded over the symmetric byte codes (Int8 / Float8\_E4M3 / Float8\_E5M2); [`scale`](./KVQuantSpec.md#scale) / [`zero_point`](./KVQuantSpec.md#zero_point) must stay at their defaults (the plane owns the scales), and [`block`](./KVQuantSpec.md#block) must stay `None`. Either way the scheme rides every tensor the cache hands out: the attention op encodes appended rows through it and decodes attended rows back.

## Data members

### dtype {#dtype}

<div class="api-signature">

[`DataType`](../../index.md#DataType) **`dtype`** `=` [`DataType::Int8`](../../index.md#DataType)

</div>

Code dtype for the ELEMENT-CODED family (Int8 / UInt8 / Float8\_E4M3 / Float8\_E5M2); ignored when [`block`](./KVQuantSpec.md#block) is set.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 137

</div>

### scale {#scale}

<div class="api-signature">

`double` **`scale`** `= 1.0`

</div>

Per-tensor static scale (element-coded family); must stay `1.0` for block-quantized and per-token schemes.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 140

</div>

### zero\_point {#zero_point}

<div class="api-signature">

`std::optional<int32_t>` **`zero_point`**

</div>

integer codes only

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 141

</div>

### block {#block}

<div class="api-signature">

[`KVBlockScheme`](./index.md#KVBlockScheme) **`block`** `=` [`KVBlockScheme::None`](./index.md#KVBlockScheme)

</div>

Block-quantized scheme; `None` (default) selects the element-coded family above.

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 144

</div>

### per\_token {#per_token}

<div class="api-signature">

`bool` **`per_token`** `= false`

</div>

online per-row scales (element-coded only)

<div class="api-declared">

Declared in `ClikaRT/nn/kv_cache.h`, line 145

</div>