ClikaRT::runtime

namespace

Classes

Name	Description
BatchableState	Mix-in for a ModelState that the continuous batcher can pack. The batcher reads these to decide how many sessions fit in one step; both must be host-cheap (no device round-trip). The getters are noexcept: an override that raises terminates in your own program rather than unwinding into the runtime, and the compiler refuses an override that is not noexcept.
BatcherConfig	Continuous-batch composition knobs.
ContainedModel	The runtime drives a Model through these contained entry points. You never build this; Executor::create / Pipeline::create build it from your Model (see detail::contain), wrapping each override in caller-side throw containment so a thrown ClikaRT::Error becomes a failure Result that the engine fails the session with; it never unwinds into the engine. (Same mechanism as http's detail::contain and the RequestCallbacks forwarder: the catch is compiled in your translation unit, only data crosses the boundary.)
ContainedStep	A PipelineStep with its Model already contained (see ContainedModel). You never build this; Pipeline::create builds one per step from your PipelineSteps, containing each step's Model in your translation unit.
ExecConfig	One executor/node's configuration.
Executor	Drives ONE Model: admits requests, runs its phases (continuous-batched across sessions), and delivers results.
FunctionModel	A concrete Model whose phases are served by callables instead of overrides.
Model	The serving-model contract: subclass this to run under Executor / Pipeline.
ModelSchema	A model's full I/O contract.
ModelState	Base for a model's per-session state. Subclass it to hold whatever a session needs across phases/steps; override as_batchable() if the state is batchable. The runtime owns the lifetime: Model::make_state returns one, the runtime threads it into Phase_RunOnce/Phase_Iterate, then calls Model::evict and destroys it.
PhaseContext	Context for a RunOnce phase (Model::Phase_RunOnce). Read inputs, write outputs; state is the session's state (null for a stateless model). All pointers are valid only for the duration of the call.
PhaseSpec	One declared phase of a model. name is borrowed; it must point into storage the model keeps alive (e.g. a static string or a member), matching the internal "model owns the name" contract.
Pipeline	A multi-step serving ensemble: each step wraps a Model with its own executor tuning, wired by input/output name maps into one request/response surface.
PipelineStep	One node of a Pipeline: a caller-owned Model* (non-owning; you keep it alive for the pipeline's lifetime) plus its config and the name remaps that wire this node's I/O to the pipeline's tensor names.
Request	A request: named input tensors plus optional scheduling hints.
RequestCallbacks	Event-driven completion hooks (consumer callbacks; they are CONTAINED at the boundary, so a throw from one never unwinds into the engine). With these set, the runtime pushes results on its pump thread and no blocking await is needed.
Response	The final response for a session.
ResponseChunk	One streamed chunk of a session's output. final == true marks the last chunk of a cleanly finishing session's stream. Driving a model directly, that chunk carries the finishing step's outputs when it produced any (a finish with no outputs delivers an empty final marker instead); a Pipeline's stream always terminates with one empty final marker, the payload rides on_complete. An erroring or cancelled session never marks a chunk final: on_complete (or the blocking retrieval) is the authoritative terminal signal in every case.
SchedulerConfig	Admission policy knob: how many sessions may be active concurrently.
SessionStepResult	The per-session result of one Iterate step (positional with the batch's slots).
StepBatchOut	The batched result of Model::Phase_Iterate: one SessionStepResult per input slot, in slot order.
StepContext	Context for an Iterate step (Model::Phase_Iterate): one entry in slots per active session in the batch. Valid only for the duration of the call.
StepSlot	One session's slot within a batched Iterate step.

Enumerations

enum Disposition

enum class Disposition : std::uint8_t

Per-session outcome of an Iterate step.

Enumerator	Description
Continue	keep stepping this session
Finish	this session is done (its outputs are final)
Error	this session failed (see SessionStepResult::error)

Declared in ClikaRT/runtime/context.h, line 23

enum PhaseKind

enum class PhaseKind : std::uint8_t

How a phase runs: RunOnce executes a single pass (e.g. prefill); Iterate is stepped repeatedly by the runtime until each session finishes (e.g. decode).

Enumerator	Description
RunOnce
Iterate

Declared in ClikaRT/runtime/schema.h, line 28

Type aliases

using SessionId

using SessionId = std::uint64_t

Opaque handle to one in-flight session, returned by enqueue.

Declared in ClikaRT/runtime/serving.h, line 24

Functions

from_graph()

FunctionModel from_graph(graph::ModelGraph graph, spec::KVSpecOptions kv = spec::KVSpecOptions())

Wrap graph as a serving node. Raises ClikaRT::Error on a graph the node cannot serve (an empty/moved-from graph; a KV graph without a positive kv capacity or whose past inputs declare no dynamic sequence axis). kv is read only for a KV-stateful graph; a stateless graph ignores it.

Declared in ClikaRT/runtime/from_graph.h, line 145

ClikaRT/runtime/context.h

#include <ClikaRT/runtime/context.h>

The context objects a runtime::Model's overrides receive and produce. They are lightweight VIEWS built by the runtime for the duration of one override call, borrowed handles (like http::ServerRequest): read them during the call, do NOT store them or the pointers/spans they expose past the call.

ClikaRT/runtime/from_graph.h

#include <ClikaRT/runtime/from_graph.h>

runtime::from_graph: serve an executable graph::ModelGraph as a pipeline node: it returns a ready-made FunctionModel whose schema derives from the graph's own input/output specs, so the node drops into a Pipeline::create step list wherever a hand-written model stood, passing every create-time gate identically.

Vocabulary: …Model = is a serving node; Model… = an artifact of a model. A graph::ModelGraph is the artifact; from_graph wraps it into the node.

graph::ModelGraph graph = graph::trace(fn, signature); // or take_graph() runtime::FunctionModel node = runtime::from_graph(std::move(graph)); auto pipe = runtime::Pipeline::create({{"node", &node, {}, {}, out_map}}, external_schema);

The graph is taken BY VALUE; the node owns it for its lifetime. A graph::CompiledFunction contributes its graph through take_graph() after one healthy call (a function that has not captured yet has no I/O specs for the pipeline's create-time gates to read).

Stateless graphs (no KV cache)

One RunOnce phase ("run"): the request's inputs are validated and mapped onto the graph's inputs BY NAME, the graph runs, and every graph output is published under its own name. A request missing a graph input raises a readable ClikaRT::Error naming the input.

KV-stateful graphs (kv_cache_info() non-empty)

The stateful arm serves graphs whose past planes are RUNTIME INPUTS (the compiled-model class): each session allocates one cache plane per past/present pair, shaped from the graph's own past-input specs with the batch and the sequence capacity taken from kv (pass spec::KVSpecOptions(batch, capacity); such a graph refuses a zero capacity). A graph whose cache planes were captured as live tensors instead (surfaced present outputs with no past inputs) carries nothing a session plane could feed, so it serves through the stateless arm and its presents emit as ordinary outputs. An "ingest" phase stashes the request's data inputs; each "step" runs the graph with every past input fed AS A VIEW of its session plane (the valid prefix, zero-length before anything is cached) and lands every present output back into its plane, advancing the session's cache cursor. Cache planes never appear in the node's schema; past inputs and present outputs are session state, not request I/O. Scope: each session runs the graph once ("step" finishes after one pass); a generative feedback loop (next step's tokens from this step's outputs, a stop condition) is a decoder node's contract, not a graph adapter's; presents land in the planes through one device copy per step. A session's planes release with its state after the runtime evicts the session.

Header-only and caller-side: everything here, including any raise, compiles into your translation unit, never the library.

ClikaRT/runtime/model.h

#include <ClikaRT/runtime/model.h>

ClikaRT::runtime::Model: the base a consumer subclasses to author a node in a serving pipeline (a tokenizer, an autoregressive decoder, a detokenizer). It is the runtime analogue of nn::Module's override-seam: you declare a schema() + phases(), then implement Phase_RunOnce (a RunOnce pass) and, for a stepped model, make_state / Phase_Iterate / evict (the Iterate loop). The runtime drives these; you never call them yourself.

Authoring a leaf node: class MyNode : public runtime::Model { const ModelSchema& schema() const override { return schema_; } Span<const PhaseSpec> phases() const override { return phases_; } void Phase_RunOnce(const PhaseSpec&, PhaseContext& ctx) override { ... } }; Override methods MAY raise ClikaRT::Error on failure; the runtime contains the throw at the boundary and fails the session cleanly (it never unwinds into the engine). A stateful model also subclasses ModelState (see state.h).

ClikaRT/runtime/pipeline.h

#include <ClikaRT/runtime/pipeline.h>

ClikaRT::runtime::Pipeline: an ensemble of Model nodes (each its own continuous-batched executor) wired into one serving graph: a request flows node→node (e.g. Tokenizer → Decoder → Detokenizer), the pipeline routes each node's outputs to the next by name. This is the primary serving entry: build it from PipelineSteps, then enqueue requests and await (or stream via RequestCallbacks). Opaque handle (move-only). You keep every Model* referenced by the steps alive for the pipeline's lifetime.

ClikaRT/runtime/schema.h

#include <ClikaRT/runtime/schema.h>

A runtime::Model's I/O contract (ModelSchema of TensorSpecs) and its phase declarations (PhaseSpec of a PhaseKind). The runtime reads these to validate requests, route tensors between pipeline nodes, and drive each phase.

ClikaRT/runtime/serving.h

#include <ClikaRT/runtime/serving.h>

The request/response vocabulary and the executor/pipeline configuration a consumer drives the runtime with. All carry only public types.

ClikaRT/runtime/state.h

#include <ClikaRT/runtime/state.h>

Per-session state bases for a stateful runtime::Model. A model that carries per-request state (a KV cache, a generation cursor) subclasses ModelState; the runtime owns one instance per live session (created by Model::make_state, destroyed after Model::evict). A state that also wants to participate in continuous batching additionally subclasses BatchableState so the batcher can pack heterogeneous-length sessions by reading occupancy() / capacity(), without naming the concrete type.