Skip to main content

ClikaRT::nn::KVQuantSpec

struct

Header: ClikaRT/nn/kv_cache.h

Per-side KV quantization scheme (quantize-on-append). Three families:

  • ELEMENT-CODED (block==KVBlockScheme::None, the default): the side stores CODE bytes at dtype (Int8 / UInt8 / Float8_E4M3 / Float8_E5M2) under the per-tensor scale (+ optional integer zero_point; fp8 codes are scale-only; one there is rejected at construction).
  • BLOCK-QUANTIZED (block set): the side stores packed blocks with inline scales (see KVBlockScheme); dtype / scale / zero_point must stay at their defaults.
  • PER-TOKEN (per_token true): the online scheme; every appended row encodes with its own absmax-derived scale, stored in a scale plane the cache allocates beside its buffer. Element-coded over the symmetric byte codes (Int8 / Float8_E4M3 / Float8_E5M2); scale / zero_point must stay at their defaults (the plane owns the scales), and block must stay None. Either way the scheme rides every tensor the cache hands out: the attention op encodes appended rows through it and decodes attended rows back.

Data members​

dtype​

DataType dtype = DataType::Int8

Code dtype for the ELEMENT-CODED family (Int8 / UInt8 / Float8_E4M3 / Float8_E5M2); ignored when block is set.

Declared in ClikaRT/nn/kv_cache.h, line 129

scale​

double scale = 1.0

Per-tensor static scale (element-coded family); must stay 1.0 for block-quantized and per-token schemes.

Declared in ClikaRT/nn/kv_cache.h, line 132

zero_point​

std::optional<int32_t> zero_point

integer codes only

Declared in ClikaRT/nn/kv_cache.h, line 133

block​

KVBlockScheme block = KVBlockScheme::None

Block-quantized scheme; None (default) selects the element-coded family above.

Declared in ClikaRT/nn/kv_cache.h, line 136

per_token​

bool per_token = false

online per-row scales (element-coded only)

Declared in ClikaRT/nn/kv_cache.h, line 137