ClikaRT::nn

namespace

Classes

Name	Description
Conv	N-dimensional convolution module (1-D/2-D/3-D by the weight's rank).
ConvTranspose	N-dimensional transposed-convolution module (learnable upsampling).
Embedding	Embedding-table module: gathers rows of its table by token index, the module form of ops::embedding.
HookOutcome	What one call of a custom operator's output_shapes or compute reported back to the runtime: Status::Ok with empty text on success, otherwise the status and text of the exception the hook raised. Plain data on purpose: an exception raised in application code is caught by the application's own C++ runtime (the entry points of detail::ModuleHookTable) and only this record crosses into the library. Not something a module author fills in by hand: the entry points in detail::ModuleHookTable produce it.
KVCache	The serving-side KV cache: per-layer storage rows (attention K/V, conv windows, recurrent slabs) over a slot-indexed batch.
KVCacheConfig	Construction parameters shared by every cache strategy. num_kv_heads is the KV head count (grouped-query attention: KV heads ≤ query heads); the cache stores K/V as [seq_len, num_kv_heads, head_dim] per (slot, layer).
KVLayerSpec	One decoder layer's storage row, the per-layer entry of the spec vector KVCache::make consumes. Two orthogonal axes: Kind says WHAT the row stores, Layout says HOW a token-indexed row is laid out.
KVQuantSpec	Per-side KV quantization scheme (quantize-on-append). Three families:ELEMENT-CODED (block == KVBlockScheme::None, the default): the side stores CODE bytes at dtype (Int8 / UInt8 / Float8_E4M3 / Float8_E5M2) under the per-tensor scale (+ optional integer zero_point; fp8 codes are scale-only; one there is rejected at construction). BLOCK-QUANTIZED (block set): the side stores packed blocks with inline scales (see KVBlockScheme); dtype / scale / zero_point must stay at their defaults. PER-TOKEN (per_token true): the online scheme; every appended row encodes with its own absmax-derived scale, stored in a scale plane the cache allocates beside its buffer. Element-coded over the symmetric byte codes (Int8 / Float8_E4M3 / Float8_E5M2); scale / zero_point must stay at their defaults (the plane owns the scales), and block must stay None. Either way the scheme rides every tensor the cache hands out: the attention op encodes appended rows through it and decodes attended rows back.
LayerNorm	Layer normalization module.
Linear	Affine transformation of the trailing dimension: $y = x W^\top + b$.
LinearOptions	Construction options: everything beyond the two feature counts. Every field is optional / defaulted; the defaults produce a plain packing Linear.
LoadOptions	Options for the load_state_dict overloads (a plain aggregate; assign the fields you need). The same options serve BOTH load paths (an in-memory NamedTensors dict and a lazy checkpoint container), so a model accounts its weight names identically whichever way it loads. strict gates the completeness report; the allow-lists carve per-name exceptions out of it without turning the whole check off:allow_missing: declared slot names the checkpoint may legitimately omit, e.g. a tied head's lm_head.weight, absent from checkpoints that share it with the embedding and bound by weight-sharing instead. allow_unexpected: checkpoint names no slot declares that are fine to leave unread, e.g. a checkpoint shipping a redundant copy of a tied head's weight. weight_residency: a load-scope residency override. When set, every module in the tree that admits a residency choice adopts this value before the checkpoint binds, as if it had been constructed with it, so one load call pins the whole model's posture without touching the model's own construction defaults. A module whose class pins its residency keeps the pin; the load logs one summary line with how many modules adopted the override and how many kept a class-pinned residency (Linear::residency() reports any one module's outcome precisely). Packed on a weight whose layout cannot pack still refuses readably at bind, exactly as a constructed Packed does. Absent (the default) changes nothing.
Module	Base class of every ClikaRT module: an ownership tree of parameters, buffers, and child modules.
ModuleHooks	The entry points a module captures when it is constructed, one per virtual the runtime invokes from its own frames (see Module::Module() and detail::ModuleHookTable). A Result-returning hook reports a caught exception as the failed Result; the others report it as a HookOutcome. Filled by Module(); a module author never constructs one.
MoE	Dense mixture-of-experts module: a router picks top-k experts per token, each expert runs its gated FFN, and the outputs combine under the router weights.
MoeOptions	MoE configuration beyond the required top-k. Every field is optional; the defaults are the runtime's (SoftmaxTopK routing, renormalized top-k weights, SwiGLU with the Interleaved gate‖up layout, erf gelu).
OutputQuant	Static output quantization for a Linear's product: requantize the output with scale / zero_point along axis, at out_dtype.
PagedParams	Shared block-pool geometry for the cache's paged rows. block_size must be a power of two in [16, 512] (a kernel addressing contract); num_blocks is the pool capacity; max_blocks_per_seq caps one sequence's block-table row.
QConv	Statically-quantized convolution module: quantized weight AND activation-quantized compute where the backend serves the scheme, Conv's static-quant sibling (construct with a QTensor weight).
QConvWoQ	Weight-only-quantized convolution module: the weight rests quantized (a QTensor) and decodes inside the kernels; compute runs at the activation dtype, Conv's weight-only sibling.
QLinear	Statically-quantized linear module: quantized weight AND activation-quantized compute where the backend serves the scheme, Linear's static-quant sibling (same forward surface).
QLinearWoQ	Weight-only-quantized linear module: the weight rests quantized (a QTensor) and decodes inside the matmul kernels; compute runs at the activation dtype, Linear's weight-only sibling (same forward surface).
QMoEWoQ	Weight-only-quantized mixture-of-experts module: the stacked [E, ...] expert weights rest quantized and decode inside the per-expert kernels, MoE's weight-only sibling (same routing surface and options).
RMSNorm	Root-mean-square normalization module.

Enumerations

enum KVBlockScheme

enum class KVBlockScheme : uint8_t

Block-quantized KV storage selector: the block-wise online schemes whose scales ride INLINE in the cache's own block bytes (no external scale to supply; the append encodes each block as it writes). Q8_0 stores 8-bit codes + one f16 scale per 32 elements (8.5 bits/element at rest); NVFP4 stores 4-bit float codes + one UE4M3 sub-scale per 16 elements over 64-element blocks (4.5 bits/element at rest). head_dim must be a whole number of the scheme's blocks. The wider block-scheme taxonomy is reachable by name through KVCacheConfig::kv_cache_scheme; this enum names only the common selections.

Enumerator	Description
None	element-coded (the dtype/scale fields below apply)
Q8_0	32-element blocks, int8 codes, inline f16 scales
NVFP4	64-element blocks, fp4 codes, inline per-16 UE4M3 sub-scales

Declared in ClikaRT/nn/kv_cache.h, line 102

enum KVCacheMode

enum class KVCacheMode : uint8_t

The cache's serving mode: what every KVLayerSpec::Layout::Auto row resolves to. A composition with no token-indexed rows serves identically under both (the mode describes the cache's token-indexed rows; a state-only stack has nothing to page).

Enumerator	Description
Continuous	per-slot buffers (preallocated or grow-on-demand)
Paged	token-indexed rows share the cache's block pool

Declared in ClikaRT/nn/kv_cache.h, line 144

enum LinearKind

enum class LinearKind : std::uint8_t

The SERVING POSTURE of a Linear-family module: what the bound weight is and how forward serves it. Reported by Linear::kind(); the runtime is built without RTTI, so this type tag is the one sanctioned runtime introspection. It names the posture, NOT the class identity: a plain Linear reports Dense until a quantized checkpoint payload binds onto its declared slot, then WeightQuantized; the subclasses report their pinned posture constantly.

Enumerator	Description
Dense	dense weight, dense matmul
WeightQuantized	quantized-at-rest weight, float activations
StaticQuant	quantized weight AND quantized activations (QLinear)

Declared in ClikaRT/nn/linear.h, line 91

enum WeightResidency

enum class WeightResidency : std::uint8_t

How a bound weight lives at rest.

Enumerator	Description
Packed	Pack once into the backend kernel layout; the pack is the resident copy.
Borrowed	Never pack a private copy; forward serves off the shared storage. A derived runtime form the call needs (e.g. a quantized weight's compute form) is retained once when it fits the device's free-memory budget, and rebuilt per call when it does not.
Auto	Pack only when the copy fits the device's free-memory budget.
PerCall	Never retain a derived runtime form on this module, the memory-pressure fallback made deterministic: forward serves off the shared storage and rebuilds any derived form per call, regardless of the free-memory budget. Minimum per-module retention at a per-call rebuild cost; the runtime may still serve repeated calls from its own bounded, process-wide working set, which this choice does not pin.

Declared in ClikaRT/nn/weight_residency.h, line 15

Variables

kAutoPackMaxFreeFraction

double kAutoPackMaxFreeFraction = 0.10

Auto packs when the head copy stays at or under this share of the device's currently-free memory: small enough that downstream pool sizing (which claims its own fraction of what remains after load) barely notices, large enough to admit every real vocab table on a workstation card. A device that reports no free-memory figure resolves to Borrowed, never a blind pack. The resolution is logged either way.

Declared in ClikaRT/nn/linear.h, line 82

ClikaRT/nn/kv_cache.h

#include <ClikaRT/nn/kv_cache.h>

A key/value cache for autoregressive decoding, the public face of the runtime's IKVCache. It holds per-layer K/V history across decode steps so attention reads the whole context while each step only computes the new tokens. Bind a layer's keys(l)/values(l) straight into ops::group_query_attention_varlen as BOTH past_key/past_value AND out_present_key/out_present_value; the op then appends this step's post-RoPE K/V into the cache buffer IN PLACE (no realloc, no copy), which is the mechanism that makes per-token decode fast.

One cache serves every decoder composition through a per-layer spec vector, on two orthogonal axes:

• Kind, WHAT a row stores: token-indexed K/V (AttentionKV, full or windowed via window), or fixed-size serving state (ConvState, RecurrentState, HybridState: slot-resident slabs the recurrence ops bind directly; no token indexing, so layout does not apply).
• Layout, HOW a token-indexed row is laid out: Continuous (a per-slot buffer, preallocated or grow-on-demand) or Paged (blocks from the cache's shared pool, handed out by reserve / prepare_step). Auto (the default) follows the cache's serving mode, so one spec vector serves both serving shapes unchanged.

Construction is ONE call: KVCache::make(config, layer_specs); the config carries the cache-wide defaults, the serving mode, and (when any row lays out paged) the shared pool geometry in config.paged. Move-only.

Prefix caching (paged rows): the contract

A cache with paged rows can reuse the K/V of already-seen prompt prefixes, so a repeated prefix (a system prompt, a chat transcript resubmitted next turn) skips its prefill compute entirely. The rules:

• Opt-in, per session. Caching happens ONLY through two calls: admit(slot, prompt_ids) at session start (binds the longest already-cached prefix) and retire(slot, full_ids) at clean finish (keeps the session's blocks addressable for future admits). Skip both and the paged cache does no hashing and no caching, zero overhead.
• Opt-out, two levels. Per session: finish via evict_batch (the cancel/error path; it drops everything, caches nothing). Entirely: never call admit/retire.
• Memory-neutral. The pool's footprint is fixed at construction (num_blocks × block_size × paged rows × heads × head_dim × dtype); caching never allocates beyond it and never pins: a cached block is FREE memory that happens to retain its bytes, reclaimed least-recently-used the moment a live sequence needs a block. A cache entry can therefore disappear under pool pressure; admit reports fewer (or zero) cached tokens.
• What it saves is prefill compute, not storage. admit returns num_cached; forward only prompt_ids[num_cached:]. Concurrent sessions sharing a prefix also share the physical blocks (copy-on-write on divergence), which REDUCES live block usage.
• State rows ride along. On a cache mixing paged rows with state rows, retire snapshots each state row's block-boundary checkpoint keyed by the prefix content hash, and admit adopts a cached prefix only when a snapshot matches the hit length exactly; otherwise it binds nothing and returns 0 (shared K/V over the wrong recurrent state is never served).
• Controls. num_blocks bounds how much history can stay cached opportunistically (size it above the live working set to leave cache headroom); block_size sets hit granularity (a hit needs a full identical block; smaller blocks match finer, at more block-table rows); retire vs evict_batch decides per session what enters the cache.
• Multimodal sessions declare their media boundary. Sharing is keyed on token ids, and the K/V under a media placeholder comes from NON-token inputs (pixels, audio) the ids cannot identify, so a session whose prompt carries media passes its FIRST media position as addressable_len on admit AND retire. Caching then covers only the pure-text prefix before it: blocks at or past the boundary are never shared or kept, and text-only sessions (the default) pay nothing.

ClikaRT/nn/linear.h

#include <ClikaRT/nn/linear.h>

Linear: the public face of the runtime's packed matmul family, exposed as an nn::Module leaf and the BASE of its posture hierarchy (QLinearWoQ pins a quantized-at-rest weight; QLinear adds static output quantization). Unlike the stateless ops::matmul / ops::linear free functions (which re-pack the weight on every call), a Linear binds its weight ONCE and packs it into the backend's kernel layout; every forward reuses that packed weight. This is the pack-once path that makes per-token decode fast; build one Linear per projection at load, reuse it every step.

Construction is ONE factory with two symmetric overloads, and the factory IS the posture dispatch:

• make(weight[, bias][, options]): construct FROM tensors: geometry, dtype and device all read off the weight; declared and bound in one call. The returned shared_ptr<Linear>'s dynamic type follows what was given: options.output_quant set ⇒ a QLinear (this wins even over a quantized weight; quantized weight + output requant IS the static-quant posture); else weight.is_quantized() ⇒ a QLinearWoQ; else a plain Linear. Model code holds the base pointer and never branches.
• make(in_features, out_features[, options]): declare-then-bind for checkpoint flows: storage-free weight / bias slots are declared under their canonical names (bias presence rides options.bias; placement rides options.device); set_weights (tensors in hand) or load_state_dict (by dotted name) binds them, and the first forward packs (once, thread-safe). options.output_quant set returns a QLinear here too. Otherwise the object is a plain Linear whose SERVING POSTURE follows the payload that later binds: a dense payload serves through the dense matmul, a quantized one through the weight-only-quantized matmul; kind() reports which. initialize() remains the optional warm-up that pays the pack at load time.

Dtype: adoption over declaration. A declared slot ADOPTS the bound payload's dtype (a weight keeps its checkpoint dtype, never a cast at rest). Until a payload binds, an unpinned slot HAS no dtype: it enumerates through named_parameters() as DataType::Undefined. LinearOptions::dtype PINS a dtype instead: the slots declare at it and every bind casts the payload to it. to(dtype), called before or after any bind, casts the slots and pins that dtype for every subsequent re-bind. One seam to know: a checkpoint RE-load (load_state_dict onto a module whose slot already holds a payload) re-binds at the slot's CURRENT dtype; re-bind through set_weights when adoption of the new payload's dtype is wanted.

Weight orientation: the canonical dense layout is the HuggingFace [out_features, in_features]. set_weights also accepts the transposed [in_features, out_features], validated against the declared feature counts (the ambiguous square case is read as [out, in]). A QUANTIZED payload may likewise carry its logical shape in either order; the pack resolves the orientation against the declared counts (square defaults to the loader's [in, out] on the weight-quantized posture). Held via std::shared_ptr (an nn::Module leaf); copy/move are pinned by the base.

ClikaRT/nn/module.h

#include <ClikaRT/nn/module.h>

ClikaRT::nn::Module: the PyTorch-nn.Module-style base every model and layer subclasses. It is BOTH the parameter/submodule container (so named_parameters() produces the same canonical dotted names, in registration order, as PyTorch, i.e. the keys in a HuggingFace model.safetensors) AND the user-space custom-op seam: a leaf module overrides output_shapes() + compute() with its own kernel logic and calls dispatch(), which runs that kernel through the runtime (eager and tracing alike).

Authoring:

• Composite: subclass, register_module(...) / register_parameter(...) in the ctor, and define your own forward(...) (any signature) composing ops:: / child modules.
• Leaf custom op: subclass, override output_shapes() + compute(), and have forward(...) call this->dispatch({inputs...}). Modules are held via std::shared_ptr (register_module stores one); a leaf that calls dispatch() must itself be owned by a shared_ptr.

ClikaRT/nn/moe.h

#include <ClikaRT/nn/moe.h>

A bound, weight-packing Mixture-of-Experts block over DENSE expert weights, the public face of the runtime's fused MoE (router top-k selection + per-expert gated FFN + the weighted combine, in ONE op). Like Linear vs ops::matmul, a MoE binds its stacked expert weights ONCE and packs them into the backend's kernel layout; every forward reuses the pack. Build one per MoE layer at load, reuse it every step. Weight-only-QUANTIZED experts are the sibling module QMoEWoQ (exactly the Linear vs QLinearWoQ split).

The lifecycle (uniform across every weight-bearing nn module):

1. make(experts, hidden, intermediate, top_k, options, ...), the ONE constructor: declares the storage-free expert / bias slots.
2. set_weights(...) or load_state_dict(...) binds them.
3. forward(x, router_logits) packs on first call (once, thread-safe), then serves. initialize() remains available as an optional warm-up.

The caller computes the per-token router logits itself (families differ in how: a plain projection, a normalized/scaled one) and hands them to forward beside the activations; routing (MoeRouting), the gated activation, and the gate‖up layout (SwigluFusion) are configuration.

Expert weights are stacked 3-D tensors (the HF "experts as one parameter" layout): gate_up_experts [E, F·I, H] (F=2 fused gate‖up, F=1 with a separate gate_experts) and down_experts [E, H, I].

ClikaRT/nn/weight_residency.h

#include <ClikaRT/nn/weight_residency.h>

ClikaRT::nn::WeightResidency: how a bound weight lives at rest. The vocabulary is shared by the module constructors that accept a residency choice (LinearOptions::residency) and by the load-scope override (nn::LoadOptions::weight_residency), so one enum names the policy wherever it is decided.