ClikaRT::runtime

namespace

Classes

Name	Description
BatchableState	Mix-in for a ModelState that the continuous batcher can pack. The batcher reads these to decide how many sessions fit in one step; both must be host-cheap (no device round-trip). The getters are noexcept: an override that raises terminates in your own program rather than unwinding into the runtime, and the compiler refuses an override that is not noexcept.
BatcherConfig	Continuous-batch composition knobs.
ContainedModel	The runtime drives a Model through these contained entry points. You never build this; Executor::create / Pipeline::create build it from your Model (see detail::contain), wrapping each override in caller-side throw containment so a thrown ClikaRT::Error becomes a failure Result that the engine fails the session with; it never unwinds into the engine. (Same mechanism as http's detail::contain and the RequestCallbacks forwarder: the catch is compiled in your translation unit, only data crosses the boundary.)
ContainedStep	A PipelineStep with its Model already contained (see ContainedModel). You never build this; Pipeline::create builds one per step from your PipelineSteps, containing each step's Model in your translation unit.
ExecConfig	One executor/node's configuration.
Executor	Drives ONE Model: admits requests, runs its phases (continuous-batched across sessions), and delivers results.
FunctionModel	A concrete Model whose phases are served by callables instead of overrides.
Model	The serving-model contract: subclass this to run under Executor / Pipeline.
ModelSchema	A model's full I/O contract.
ModelState	Base for a model's per-session state. Subclass it to hold whatever a session needs across phases/steps; override as_batchable() if the state is batchable. The runtime owns the lifetime: Model::make_state returns one, the runtime threads it into Phase_RunOnce/Phase_Iterate, then calls Model::evict and destroys it.
PhaseContext	Context for a RunOnce phase (Model::Phase_RunOnce). Read inputs, write outputs; state is the session's state (null for a stateless model). All pointers are valid only for the duration of the call.
PhaseSpec	One declared phase of a model. name is borrowed; it must point into storage the model keeps alive (e.g. a static string or a member), matching the internal "model owns the name" contract.
Pipeline	A multi-step serving ensemble: each step wraps a Model with its own executor tuning, wired by input/output name maps into one request/response surface.
PipelineStep	One node of a Pipeline: a caller-owned Model* (non-owning; you keep it alive for the pipeline's lifetime) plus its config and the name remaps that wire this node's I/O to the pipeline's tensor names.
Request	A request: named input tensors plus optional scheduling hints.
RequestCallbacks	Event-driven completion hooks (consumer callbacks; they are CONTAINED at the boundary, so a throw from one never unwinds into the engine). With these set, the runtime pushes results on its pump thread and no blocking await is needed.
Response	The final response for a session.
ResponseChunk	One streamed chunk of a session's output. final == true marks the last chunk of a cleanly finishing session's stream. Driving a model directly, that chunk carries the finishing step's outputs when it produced any (a finish with no outputs delivers an empty final marker instead); a Pipeline's stream always terminates with one empty final marker, the payload rides on_complete. An erroring or cancelled session never marks a chunk final: on_complete (or the blocking retrieval) is the authoritative terminal signal in every case.
SchedulerConfig	Admission policy knobs: how many sessions may be active concurrently, and how admission accounts for the model's capacity.
SessionStepResult	The per-session result of one Iterate step (positional with the batch's slots).
StepBatchOut	The batched result of Model::Phase_Iterate: one SessionStepResult per input slot, in slot order.
StepContext	Context for an Iterate step (Model::Phase_Iterate): one entry in slots per active session in the batch. Valid only for the duration of the call.
StepSlot	One session's slot within a batched Iterate step. session and priority let a model short of capacity pick which session to set aside without the runtime naming one: the youngest (highest session; ids are minted in arrival order) among the lowest priority. See Disposition::Preempted.
UnitProjection	A request's need in the model's own capacity units (see Model::capacity_units), as Model::project_units reads it from the request's inputs before admission. -1 means unknown: the request is then admitted under the session cap alone.

Enumerations

enum Disposition

enum class Disposition : std::uint8_t

Per-session outcome of an Iterate step.

Enumerator	Description
Continue	keep stepping this session
Finish	this session is done (its outputs are final)
Error	this session failed (see SessionStepResult::error)
Preempted	This session is intact but was NOT run this step: the model ran short of capacity for the batch and set it aside (the youngest of the lowest priority, read from the StepSlot fields). The runtime releases the session's state through Model::evict, merges the result's outputs into the request's inputs as the RESUME payload (whatever the model needs to rebuild the session, under its own input names: the generated ids, the remaining output budget; nothing is delivered to the caller), and re-admits the session at the head of the queue. Its earlier chunks stay delivered, so the stream neither repeats nor drops one. Never an error: the request is served, later. A model sets a session aside only when another session can take the room; the last session standing that outgrows the capacity reports Error instead. Only CapacityPolicy::PreemptLast asks a model for this verdict (see Model::capacity_units).

Declared in ClikaRT/runtime/context.h, line 23

enum PhaseKind

enum class PhaseKind : std::uint8_t

How a phase runs: RunOnce executes a single pass (e.g. prefill); Iterate is stepped repeatedly by the runtime until each session finishes (e.g. decode).

Enumerator	Description
RunOnce
Iterate

Declared in ClikaRT/runtime/schema.h, line 28

enum CapacityPolicy

enum class CapacityPolicy : std::uint8_t

How admission accounts for the model's capacity beyond the session cap. The units are the model's own (Model::capacity_units / Model::project_units: tokens of context for a language model, frames for a streaming encoder); the runtime compares the integers and never interprets them. A model that reports no capacity (capacity_units() < 0) is admitted under SlotCap whatever the policy says. Under NoEvict and PreemptLast, a request that does not fit even an idle model fails at admission with Status::InvalidArgument, naming its need and the model's capacity in the model's units, instead of failing at a step.

Enumerator	Description
SlotCap	Admit up to max_active_sessions; the model's capacity is its own affair (a request the model cannot hold fails inside the model). The default; the model's capacity is never read.
NoEvict	Admit a request only while the free capacity covers every active session's remaining projection plus this request's whole projection, so an admitted session never runs out of room. Admission is predictable; the cost is capacity held for output that may never be produced.
PreemptLast	Admit while the free capacity covers the active sessions' next step plus this request's first step. When the model runs short at a step it sets a session aside (Disposition::Preempted, the youngest of the lowest priority); the runtime releases that session's state and re-admits it at the head of the queue with its delivered chunks intact, so its stream neither repeats nor drops one.

Declared in ClikaRT/runtime/serving.h, line 79

Type aliases

using SessionId

using SessionId = std::uint64_t

Opaque handle to one in-flight session, returned by enqueue.

Declared in ClikaRT/runtime/serving.h, line 24

Functions

from_graph()

FunctionModel from_graph(graph::ModelGraph graph, spec::KVSpecOptions kv = spec::KVSpecOptions())

Wrap graph as a serving node. Raises ClikaRT::Error on a graph the node cannot serve (an empty/moved-from graph; a KV graph without a positive kv capacity or whose past inputs declare no dynamic sequence axis). kv is read only for a KV-stateful graph; a stateless graph ignores it.

Declared in ClikaRT/runtime/from_graph.h, line 145

ClikaRT/runtime/context.h

#include <ClikaRT/runtime/context.h>

The context objects a runtime::Model's overrides receive and produce. They are lightweight VIEWS built by the runtime for the duration of one override call, borrowed handles (like http::ServerRequest): read them during the call, do NOT store them or the pointers/spans they expose past the call.

ClikaRT/runtime/from_graph.h

#include <ClikaRT/runtime/from_graph.h>

runtime::from_graph: serve an executable graph::ModelGraph as a pipeline node: it returns a ready-made FunctionModel whose schema derives from the graph's own input/output specs, so the node drops into a Pipeline::create step list wherever a hand-written model stood, passing every create-time gate identically.

Vocabulary: …Model = is a serving node; Model… = an artifact of a model. A graph::ModelGraph is the artifact; from_graph wraps it into the node.

graph::ModelGraph graph = graph::trace(fn, signature); // or take_graph() runtime::FunctionModel node = runtime::from_graph(std::move(graph)); auto pipe = runtime::Pipeline::create({{"node", &node, {}, {}, out_map}}, external_schema);

The graph is taken BY VALUE; the node owns it for its lifetime. A graph::CompiledFunction contributes its graph through take_graph() after one healthy call (a function that has not captured yet has no I/O specs for the pipeline's create-time gates to read).

Stateless graphs (no KV cache)

One RunOnce phase ("run"): the request's inputs are validated and mapped onto the graph's inputs BY NAME, the graph runs, and every graph output is published under its own name. A request missing a graph input raises a readable ClikaRT::Error naming the input.

KV-stateful graphs (kv_cache_info() non-empty)

The stateful arm serves graphs whose past planes are RUNTIME INPUTS (the compiled-model class): each session allocates one cache plane per past/present pair, shaped from the graph's own past-input specs with the batch and the sequence capacity taken from kv (pass spec::KVSpecOptions(batch, capacity); such a graph refuses a zero capacity). A graph whose cache planes were captured as live tensors instead (surfaced present outputs with no past inputs) carries nothing a session plane could feed, so it serves through the stateless arm and its presents emit as ordinary outputs. An "ingest" phase stashes the request's data inputs; each "step" runs the graph with every past input fed AS A VIEW of its session plane (the valid prefix, zero-length before anything is cached) and lands every present output back into its plane, advancing the session's cache cursor. Cache planes never appear in the node's schema; past inputs and present outputs are session state, not request I/O. Scope: each session runs the graph once ("step" finishes after one pass); a generative feedback loop (next step's tokens from this step's outputs, a stop condition) is a decoder node's contract, not a graph adapter's; presents land in the planes through one device copy per step. A session's planes release with its state after the runtime evicts the session.

Header-only and caller-side: everything here, including any raise, compiles into your translation unit, never the library.

ClikaRT/runtime/model.h

#include <ClikaRT/runtime/model.h>

ClikaRT::runtime::Model: the base a consumer subclasses to author a node in a serving pipeline (a tokenizer, an autoregressive decoder, a detokenizer). It is the runtime analogue of nn::Module's override-seam: you declare a schema() + phases(), then implement Phase_RunOnce (a RunOnce pass) and, for a stepped model, make_state / Phase_Iterate / evict (the Iterate loop). The runtime drives these; you never call them yourself.

Authoring a leaf node: class MyNode : public runtime::Model { const ModelSchema& schema() const override { return schema_; } Span<const PhaseSpec> phases() const override { return phases_; } void Phase_RunOnce(const PhaseSpec&, PhaseContext& ctx) override { ... } }; Override methods MAY raise ClikaRT::Error on failure; the runtime contains the throw at the boundary and fails the session cleanly (it never unwinds into the engine). A stateful model also subclasses ModelState (see state.h).

ClikaRT/runtime/pipeline.h

#include <ClikaRT/runtime/pipeline.h>

ClikaRT::runtime::Pipeline: an ensemble of Model nodes (each its own continuous-batched executor) wired into one serving graph: a request flows node→node (e.g. Tokenizer → Decoder → Detokenizer), the pipeline routes each node's outputs to the next by name. This is the primary serving entry: build it from PipelineSteps, then enqueue requests and await (or stream via RequestCallbacks). Opaque handle (move-only). You keep every Model* referenced by the steps alive for the pipeline's lifetime.

ClikaRT/runtime/schema.h

#include <ClikaRT/runtime/schema.h>

A runtime::Model's I/O contract (ModelSchema of TensorSpecs) and its phase declarations (PhaseSpec of a PhaseKind). The runtime reads these to validate requests, route tensors between pipeline nodes, and drive each phase.

ClikaRT/runtime/serving.h

#include <ClikaRT/runtime/serving.h>

The request/response vocabulary and the executor/pipeline configuration a consumer drives the runtime with. All carry only public types.

ClikaRT/runtime/state.h

#include <ClikaRT/runtime/state.h>

Per-session state bases for a stateful runtime::Model. A model that carries per-request state (a KV cache, a generation cursor) subclasses ModelState; the runtime owns one instance per live session (created by Model::make_state, destroyed after Model::evict). A state that also wants to participate in continuous batching additionally subclasses BatchableState so the batcher can pack heterogeneous-length sessions by reading occupancy() / capacity(), without naming the concrete type.