Skip to main content

KVCache

The serving-side KV cache: per-layer storage rows over slot-indexed batches. Built once with make(); the attention ops append and attend through the tensors it hands out.

__init__​

__init__(self, /, *args, **kwargs)

Initialize self. See help(type(self)) for accurate signature.

admit​

admitadmit(self, batch_idx: int, prompt_ids: clika_runtime._core.Tensor, addressable_len: int = -1) -> int

admit(self, batch_idx: int, prompt_ids: clika_runtime._core.Tensor, addressable_len: int = -1) -> int

Paged only: bind the longest already-cached prefix into a fresh slot; returns the number of leading tokens whose K/V is already cached.

conv_state​

conv_stateconv_state(self, layer: int, index: int = 0) -> clika_runtime._core.Tensor

conv_state(self, layer: int, index: int = 0) -> clika_runtime._core.Tensor

A state layer's i-th conv window slab [max_seqs, dim, width].

evict_batch​

evict_batchevict_batch(self, batch_idx: int) -> None

evict_batch(self, batch_idx: int) -> None

Free a slot's stored K/V (its seq_lens entry returns to 0).

fill​

fillfill(self, layer: int, batch_idx: int, k: clika_runtime._core.Tensor, v: clika_runtime._core.Tensor) -> None

fill(self, layer: int, batch_idx: int, k: clika_runtime._core.Tensor, v: clika_runtime._core.Tensor) -> None

Fill a CrossAttentionKV layer's K/V planes for one slot in one call ([t_enc, kv_heads, head_dim] each).

keys​

keyskeys(self, layer: int) -> clika_runtime._core.Tensor

keys(self, layer: int) -> clika_runtime._core.Tensor

This layer's K storage view; bind as past AND present key into the varlen attention ops.

make​

make(*args, **kwargs)

make(config: clika_runtime._core.nn.KVCacheConfig, layer_specs: collections.abc.Sequence[clika_runtime._core.nn.KVLayerSpec]) -> clika_runtime._core.nn.KVCache

Build a cache with one spec row per layer (the list's length must equal config.num_layers).

mode​

modemode(self) -> clika_runtime._core.nn.KVCacheMode

mode(self) -> clika_runtime._core.nn.KVCacheMode

The serving mode this cache was built with.

paged_block_table​

paged_block_tablepaged_block_table(self) -> clika_runtime._core.Tensor

paged_block_table(self) -> clika_runtime._core.Tensor

The paged block table [max_seqs, max_blocks_per_seq] Int32, or an undefined tensor for a cache with no paged rows.

prepare_step​

prepare_stepprepare_step(self, cu_seqlens_q: collections.abc.Sequence[int], past_len: collections.abc.Sequence[int], slot_ids: collections.abc.Sequence[int]) -> clika_runtime._core.nn.StepIndices

prepare_step(self, cu_seqlens_q: collections.abc.Sequence[int], past_len: collections.abc.Sequence[int], slot_ids: collections.abc.Sequence[int]) -> clika_runtime._core.nn.StepIndices

Convert one step's host-side description into the device-side index tensors a varlen attention forward binds (a paged cache also reserves the step's blocks). One call per step.

recurrent_state​

recurrent_staterecurrent_state(self, layer: int) -> clika_runtime._core.Tensor

recurrent_state(self, layer: int) -> clika_runtime._core.Tensor

A state layer's recurrent slab [max_seqs, heads, k_dim, v_dim].

reserve​

reservereserve(self, slot_ids: clika_runtime._core.Tensor, n_new_per_seq: collections.abc.Sequence[int]) -> clika_runtime._core.Tensor

reserve(self, slot_ids: clika_runtime._core.Tensor, n_new_per_seq: collections.abc.Sequence[int]) -> clika_runtime._core.Tensor

Paged only: reserve this step's new tokens per slot; returns the per-sequence pre-advance write offsets.

reset​

resetreset(self) -> None

reset(self) -> None

Drop all stored K/V across every slot and layer.

retire​

retireretire(self, batch_idx: int, full_token_ids: clika_runtime._core.Tensor, addressable_len: int = -1) -> None

retire(self, batch_idx: int, full_token_ids: clika_runtime._core.Tensor, addressable_len: int = -1) -> None

Paged only: retire a finished session, keeping its blocks addressable for future admits; use evict_batch on the cancel/error path instead.

seq_lens​

seq_lensseq_lens(self) -> clika_runtime._core.Tensor

seq_lens(self) -> clika_runtime._core.Tensor

Per-slot effective token counts, [max_seqs] Int32 on the cache device.

update_position​

update_positionupdate_position(self, batch_idx: int, new_pos: int) -> None

update_position(self, batch_idx: int, new_pos: int) -> None

Set a slot's effective seq_len (bookkeeping only).

values​

valuesvalues(self, layer: int) -> clika_runtime._core.Tensor

values(self, layer: int) -> clika_runtime._core.Tensor

The value-side sibling of keys(layer).