Skip to main content

ClikaRT::nn::KVCacheFootprint

struct

Header: ClikaRT/nn/kv_cache.h

What KVCache::make allocates for a composition, by storage class, and the two figures a serving stack budgets beside them. Every device figure is the bytes the construction allocates for that class, read off the same per-row plan the cache builds its storage from (one law, never a second estimate); device_bytes_at_make() is their sum. A serving stack sizes its pool against device_bytes_at_steady_state() plus snapshot_host_budget_bytes on the host.

Member functions​

device_bytes_at_make()​

std::uint64_t device_bytes_at_make() const noexcept

Every device byte make allocates.

Declared in ClikaRT/nn/kv_cache.h, line 437

device_bytes_at_steady_state()​

std::uint64_t device_bytes_at_steady_state() const noexcept

The device bytes a serving cache settles at: grow-on-demand rows at their ceiling, the checkpoint shadows allocated.

Declared in ClikaRT/nn/kv_cache.h, line 442

Data members​

continuous_bytes​

std::uint64_t continuous_bytes = 0

Continuous full-attention rows (CrossAttentionKV included): both planes at their construction depth, the full max_tokens_per_seq under preallocate, a small seed on a grow-on-demand row; with each per-token side's scale plane and the row's per-slot length vector.

Declared in ClikaRT/nn/kv_cache.h, line 405

continuous_ceiling_bytes​

std::uint64_t continuous_ceiling_bytes = 0

The same rows at their full depth: what a grow-on-demand row reaches as its slots fill. Equal to continuous_bytes when every continuous row preallocates.

Declared in ClikaRT/nn/kv_cache.h, line 409

windowed_bytes​

std::uint64_t windowed_bytes = 0

Windowed rows: a continuous ring's planes (window-deep) and a windowed paged row's own pools and block table, one per distinct window value, provisioned outside PagedParams::num_blocks.

Declared in ClikaRT/nn/kv_cache.h, line 413

paged_bytes​

std::uint64_t paged_bytes = 0

The full-attention paged rows: every geometry group's pool planes over PagedParams::num_blocks blocks (a per-token side's scale planes included) and the shared block table.

Declared in ClikaRT/nn/kv_cache.h, line 417

paged_block_bytes​

std::uint64_t paged_block_bytes = 0

The bytes one block of the full-attention paged rows costs: every geometry group's pool planes for a single block, a per-token side's scale planes included; the unit a pool planner converts a byte budget into blocks with. paged_bytes is PagedParams::num_blocks of these plus the shared block table. Zero when no full-attention row is paged.

Declared in ClikaRT/nn/kv_cache.h, line 423

state_bytes​

std::uint64_t state_bytes = 0

The state rows' conv window slabs and recurrent slabs.

Declared in ClikaRT/nn/kv_cache.h, line 425

shadow_bytes​

std::uint64_t shadow_bytes = 0

The state rows' boundary-checkpoint shadows: the state slabs once more, allocated at a slot's first block-boundary checkpoint on a cache that carries paged rows (the prefix-adoption leg). Zero when no row is paged.

Declared in ClikaRT/nn/kv_cache.h, line 430

snapshot_host_budget_bytes​

std::uint64_t snapshot_host_budget_bytes = 0

HOST memory: the ceiling of the retired-state snapshot store a cache with state rows beside paged rows fills as sessions retire at block boundaries. Zero otherwise.

Declared in ClikaRT/nn/kv_cache.h, line 434