Skip to main content

ClikaRT::nn::KVCacheConfig

struct

Header: ClikaRT/nn/kv_cache.h

Construction parameters shared by every cache strategy. num_kv_heads is the KV head count (grouped-query attention: KV heads ≤ query heads); the cache stores K/V as [seq_len, num_kv_heads, head_dim] per (slot, layer).

Data members​

num_layers​

int32_t num_layers = 0

Decoder layer count (one spec row per layer).

Declared in ClikaRT/nn/kv_cache.h, line 333

num_kv_heads​

int32_t num_kv_heads = 0

Default KV head count for rows that leave KVLayerSpec::kv_heads at 0.

Declared in ClikaRT/nn/kv_cache.h, line 336

head_dim​

int32_t head_dim = 0

Default per-head width for rows that leave their head_dim fields at 0.

Declared in ClikaRT/nn/kv_cache.h, line 339

kv_dtype​

DataType kv_dtype = DataType::BFloat16

Storage dtype for unquantized K/V planes.

Declared in ClikaRT/nn/kv_cache.h, line 341

max_seqs​

int32_t max_seqs = 1

concurrent batch slots

Declared in ClikaRT/nn/kv_cache.h, line 342

max_tokens_per_seq​

int64_t max_tokens_per_seq = 0

per-slot token cap (the preallocated depth)

Declared in ClikaRT/nn/kv_cache.h, line 343

device​

Device device = Device::cpu()

Declared in ClikaRT/nn/kv_cache.h, line 344

k_quant​

std::optional<KVQuantSpec> k_quant

quantize the K side (see KVQuantSpec)

Declared in ClikaRT/nn/kv_cache.h, line 345

v_quant​

std::optional<KVQuantSpec> v_quant

quantize the V side

Declared in ClikaRT/nn/kv_cache.h, line 346

kv_cache_scheme​

std::string kv_cache_scheme

Checkpoint-convention KV quantization: the fields a quantized checkpoint's config declares, bound here so a loader passes them through instead of re-deriving typed specs:

  • kv_cache_scheme names a BLOCK storage scheme for BOTH sides by its scheme string (e.g. "GGUF_Q8_0", "NVFP4_E4M3"); the string form reaches the runtime's full block-scheme taxonomy without a wide enum. It is an alternative spelling of KVQuantSpec::block; setting both on one side raises. Empty (the default) = unused. An unknown name raises. Element-coded checkpoint schemes (e.g. a per-tensor fp8 KV cache) are expressed through the typed KVQuantSpec fields instead.
  • k_scale / v_scale are the checkpoint's per-tensor calibration scales; each feeds that side's KVQuantSpec::scale. The side must be element-coded quantized: a scale on an unquantized side, a side whose spec already carries a non-default scale, a per-token side (it derives its own row scales), or a block side (its scales ride inline; a checkpoint global there is encode-time provenance an online-encoding cache does not consume) raises rather than being dropped.

Declared in ClikaRT/nn/kv_cache.h, line 367

k_scale​

std::optional<double> k_scale

Model-level K-side calibration scale (pairs with kv_cache_scheme); absent = uncalibrated.

Declared in ClikaRT/nn/kv_cache.h, line 370

v_scale​

std::optional<double> v_scale

Model-level V-side calibration scale, k_scale's twin.

Declared in ClikaRT/nn/kv_cache.h, line 372

mode​

KVCacheMode mode = KVCacheMode::Continuous

The serving mode; it resolves every row's Layout::Auto. A composition with no token-indexed rows serves identically under both modes.

Declared in ClikaRT/nn/kv_cache.h, line 377

paged​

std::optional<PagedParams> paged

Shared block-pool geometry. Required when any row lays out Paged (via mode or an explicit Layout::Paged pin); ignored when none does. Per-layer K/V geometry (a row's own head_dim / v_head_dim) rides KVLayerSpec, not this config.

Declared in ClikaRT/nn/kv_cache.h, line 382