---
title: "ClikaRT::runtime"
sidebar_label: "runtime"
description: "Namespace ClikaRT::runtime"
---

<!-- Generated by tools/doxygen2mdx from Doxygen XML. Do not edit. -->

<span class="api-kind api-kind--namespace">namespace</span>

## Classes

| Name | Description |
| --- | --- |
| [`BatchableState`](./BatchableState.md) | Mix-in for a [`ModelState`](./ModelState.md) that the continuous batcher can pack. The batcher reads these to decide how many sessions fit in one step; both must be host-cheap (no device round-trip). The getters are `noexcept`: an override that raises terminates in your own program rather than unwinding into the runtime, and the compiler refuses an override that is not `noexcept`. |
| [`BatcherConfig`](./BatcherConfig.md) | Continuous-batch composition knobs. |
| [`ContainedModel`](./ContainedModel.md) | The runtime drives a [`Model`](./Model.md) through these contained entry points. You never build this; [`Executor::create`](./Executor.md#create) / [`Pipeline::create`](./Pipeline.md#create) build it from your [`Model`](./Model.md) (see `detail::contain`), wrapping each override in caller-side throw containment so a thrown [`ClikaRT::Error`](../Error.md) becomes a failure [`Result`](../Result.md) that the engine fails the session with; it never unwinds into the engine. (Same mechanism as [`http`](../http/index.md)'s `detail::contain` and the [`RequestCallbacks`](./RequestCallbacks.md) forwarder: the catch is compiled in your translation unit, only data crosses the boundary.) |
| [`ContainedStep`](./ContainedStep.md) | A [`PipelineStep`](./PipelineStep.md) with its [`Model`](./Model.md) already contained (see [`ContainedModel`](./ContainedModel.md)). You never build this; [`Pipeline::create`](./Pipeline.md#create) builds one per step from your [`PipelineStep`](./PipelineStep.md)s, containing each step's [`Model`](./Model.md) in your translation unit. |
| [`ExecConfig`](./ExecConfig.md) | One executor/node's configuration. |
| [`Executor`](./Executor.md) | Drives ONE [`Model`](./Model.md): admits requests, runs its phases (continuous-batched across sessions), and delivers results. |
| [`FunctionModel`](./FunctionModel.md) | A concrete [`Model`](./Model.md) whose phases are served by callables instead of overrides. |
| [`Model`](./Model.md) | The serving-model contract: subclass this to run under [`Executor`](./Executor.md) / [`Pipeline`](./Pipeline.md). |
| [`ModelSchema`](./ModelSchema.md) | A model's full I/O contract. |
| [`ModelState`](./ModelState.md) | Base for a model's per-session state. Subclass it to hold whatever a session needs across phases/steps; override [`as_batchable()`](./ModelState.md#as_batchable) if the state is batchable. The runtime owns the lifetime: [`Model::make_state`](./Model.md#make_state) returns one, the runtime threads it into `Phase_RunOnce`/`Phase_Iterate`, then calls [`Model::evict`](./Model.md#evict) and destroys it. |
| [`PhaseContext`](./PhaseContext.md) | Context for a `RunOnce` phase ([`Model::Phase_RunOnce`](./Model.md#Phase_RunOnce)). Read [`inputs`](./PhaseContext.md#inputs), write [`outputs`](./PhaseContext.md#outputs); [`state`](./PhaseContext.md#state) is the session's state (null for a stateless model). All pointers are valid only for the duration of the call. |
| [`PhaseSpec`](./PhaseSpec.md) | One declared phase of a model. [`name`](./PhaseSpec.md#name) is borrowed; it must point into storage the model keeps alive (e.g. a static string or a member), matching the internal "model owns the name" contract. |
| [`Pipeline`](./Pipeline.md) | A multi-step serving ensemble: each step wraps a [`Model`](./Model.md) with its own executor tuning, wired by input/output name maps into one request/response surface. |
| [`PipelineStep`](./PipelineStep.md) | One node of a [`Pipeline`](./Pipeline.md): a caller-owned `Model*` (non-owning; you keep it alive for the pipeline's lifetime) plus its config and the name remaps that wire this node's I/O to the pipeline's tensor names. |
| [`Request`](./Request.md) | A request: named input tensors plus optional scheduling hints. |
| [`RequestCallbacks`](./RequestCallbacks.md) | Event-driven completion hooks (consumer callbacks; they are CONTAINED at the boundary, so a throw from one never unwinds into the engine). With these set, the runtime pushes results on its pump thread and no blocking `await` is needed. |
| [`Response`](./Response.md) | The final response for a session. |
| [`ResponseChunk`](./ResponseChunk.md) | One streamed chunk of a session's output. [`final`](./ResponseChunk.md#final)` == true` marks the last chunk of a cleanly finishing session's stream. Driving a model directly, that chunk carries the finishing step's outputs when it produced any (a finish with no outputs delivers an empty final marker instead); a [`Pipeline`](./Pipeline.md)'s stream always terminates with one empty final marker, the payload rides `on_complete`. An erroring or cancelled session never marks a chunk final: `on_complete` (or the blocking retrieval) is the authoritative terminal signal in every case. |
| [`SchedulerConfig`](./SchedulerConfig.md) | Admission policy knobs: how many sessions may be active concurrently, and how admission accounts for the model's capacity. |
| [`SessionStepResult`](./SessionStepResult.md) | The per-session result of one `Iterate` step (positional with the batch's slots). |
| [`StepBatchOut`](./StepBatchOut.md) | The batched result of [`Model::Phase_Iterate`](./Model.md#Phase_Iterate): one [`SessionStepResult`](./SessionStepResult.md) per input slot, in slot order. |
| [`StepContext`](./StepContext.md) | Context for an `Iterate` step ([`Model::Phase_Iterate`](./Model.md#Phase_Iterate)): one entry in [`slots`](./StepContext.md#slots) per active session in the batch. Valid only for the duration of the call. |
| [`StepSlot`](./StepSlot.md) | One session's slot within a batched `Iterate` step. [`session`](./StepSlot.md#session) and [`priority`](./StepSlot.md#priority) let a model short of capacity pick which session to set aside without the runtime naming one: the youngest (highest [`session`](./StepSlot.md#session); ids are minted in arrival order) among the lowest [`priority`](./StepSlot.md#priority). See [`Disposition::Preempted`](./index.md#Disposition). |
| [`UnitProjection`](./UnitProjection.md) | A request's need in the model's own capacity units (see [`Model::capacity_units`](./Model.md#capacity_units)), as [`Model::project_units`](./Model.md#project_units) reads it from the request's inputs before admission. -1 means unknown: the request is then admitted under the session cap alone. |

## Enumerations

### enum Disposition {#Disposition}

<div class="api-signature">

`enum class` **`Disposition`** `:` `std::uint8_t`

</div>

Per-session outcome of an `Iterate` step.

| Enumerator | Description |
| --- | --- |
| `Continue` | keep stepping this session |
| `Finish` | this session is done (its outputs are final) |
| `Error` | this session failed (see [`SessionStepResult::error`](./SessionStepResult.md#error)) |
| `Preempted` | This session is intact but was NOT run this step: the model ran short of capacity for the batch and set it aside (the youngest of the lowest priority, read from the [`StepSlot`](./StepSlot.md) fields). The runtime releases the session's state through [`Model::evict`](./Model.md#evict), merges the result's `outputs` into the request's inputs as the RESUME payload (whatever the model needs to rebuild the session, under its own input names: the generated ids, the remaining output budget; nothing is delivered to the caller), and re-admits the session at the head of the queue. Its earlier chunks stay delivered, so the stream neither repeats nor drops one. Never an error: the request is served, later. A model sets a session aside only when another session can take the room; the last session standing that outgrows the capacity reports [`Error`](../Error.md) instead. Only [`CapacityPolicy::PreemptLast`](./index.md#CapacityPolicy) asks a model for this verdict (see [`Model::capacity_units`](./Model.md#capacity_units)). |

<div class="api-declared">

Declared in `ClikaRT/runtime/context.h`, line 23

</div>

### enum PhaseKind {#PhaseKind}

<div class="api-signature">

`enum class` **`PhaseKind`** `:` `std::uint8_t`

</div>

How a phase runs: `RunOnce` executes a single pass (e.g. prefill); `Iterate` is stepped repeatedly by the runtime until each session finishes (e.g. decode).

| Enumerator | Description |
| --- | --- |
| `RunOnce` |  |
| `Iterate` |  |

<div class="api-declared">

Declared in `ClikaRT/runtime/schema.h`, line 28

</div>

### enum CapacityPolicy {#CapacityPolicy}

<div class="api-signature">

`enum class` **`CapacityPolicy`** `:` `std::uint8_t`

</div>

How admission accounts for the model's capacity beyond the session cap. The units are the model's own ([`Model::capacity_units`](./Model.md#capacity_units) / [`Model::project_units`](./Model.md#project_units): tokens of context for a language model, frames for a streaming encoder); the runtime compares the integers and never interprets them. A model that reports no capacity (`capacity_units() < 0`) is admitted under `SlotCap` whatever the policy says. Under `NoEvict` and `PreemptLast`, a request that does not fit even an idle model fails at admission with [`Status::InvalidArgument`](../../index.md#Status), naming its need and the model's capacity in the model's units, instead of failing at a step.

| Enumerator | Description |
| --- | --- |
| `SlotCap` | Admit up to `max_active_sessions`; the model's capacity is its own affair (a request the model cannot hold fails inside the model). The default; the model's capacity is never read. |
| `NoEvict` | Admit a request only while the free capacity covers every active session's remaining projection plus this request's whole projection, so an admitted session never runs out of room. Admission is predictable; the cost is capacity held for output that may never be produced. |
| `PreemptLast` | Admit while the free capacity covers the active sessions' next step plus this request's first step. When the model runs short at a step it sets a session aside ([`Disposition::Preempted`](./index.md#Disposition), the youngest of the lowest priority); the runtime releases that session's state and re-admits it at the head of the queue with its delivered chunks intact, so its stream neither repeats nor drops one. |

<div class="api-declared">

Declared in `ClikaRT/runtime/serving.h`, line 79

</div>

## Type aliases

### using SessionId {#SessionId}

<div class="api-signature">

`using` **`SessionId`** `=` `std::uint64_t`

</div>

Opaque handle to one in-flight session, returned by `enqueue`.

<div class="api-declared">

Declared in `ClikaRT/runtime/serving.h`, line 24

</div>

## Functions

### from\_graph() {#from_graph}

<div class="api-signature">

[`FunctionModel`](./FunctionModel.md) **`from_graph`**([`graph::ModelGraph`](../graph/ModelGraph.md) `graph`, [`spec::KVSpecOptions`](../spec/KVSpecOptions.md) `kv` `=` [`spec::KVSpecOptions`](../spec/KVSpecOptions.md)`()`)

</div>

Wrap [`graph`](../graph/index.md) as a serving node. Raises [`ClikaRT::Error`](../Error.md) on a graph the node cannot serve (an empty/moved-from graph; a KV graph without a positive `kv` capacity or whose past inputs declare no dynamic sequence axis). `kv` is read only for a KV-stateful graph; a stateless graph ignores it. Hand it a finalized graph (`ModelGraph::finalize`): a graph that is not refuses its first request (`FAILED_PRECONDITION`).

<div class="api-declared">

Declared in `ClikaRT/runtime/from_graph.h`, line 155

</div>

## `ClikaRT/runtime/context.h` {#context-h}

```cpp
#include <ClikaRT/runtime/context.h>
```

The context objects a `runtime::Model`'s overrides receive and produce. They are lightweight VIEWS built by the runtime for the duration of one override call, borrowed handles (like `http::ServerRequest`): read them during the call, do NOT store them or the pointers/spans they expose past the call.

## `ClikaRT/runtime/from_graph.h` {#from_graph-h}

```cpp
#include <ClikaRT/runtime/from_graph.h>
```

`runtime::from_graph`: serve an executable `graph::ModelGraph` as a pipeline node: it returns a ready-made `FunctionModel` whose schema derives from the graph's own input/output specs, so the node drops into a `Pipeline::create` step list wherever a hand-written model stood, passing every create-time gate identically.

Vocabulary: …Model = is a serving node; Model… = an artifact of a model. A `graph::ModelGraph` is the artifact; `from_graph` wraps it into the node.

graph::ModelGraph graph = graph::trace(fn, signature); graph.optimize(); graph.finalize(); // only a finalized graph serves; take\_graph() hands one back finalized runtime::FunctionModel node = runtime::from\_graph(std::move(graph)); auto pipe = runtime::Pipeline::create(\{\{"node", \&node, \{\}, \{\}, out\_map\}\}, external\_schema);

The graph is taken BY VALUE; the node owns it for its lifetime. A `graph::CompiledFunction` contributes its graph through `take_graph()` after one healthy call (a function that has not captured yet has no I/O specs for the pipeline's create-time gates to read).

###  {#autotoc_md4_1s1}

#### Stateless graphs (no KV cache) {#autotoc_md4}

One `RunOnce` phase ("run"): the request's inputs are validated and mapped onto the graph's inputs BY NAME, the graph runs, and every graph output is published under its own name. A request missing a graph input raises a readable [`ClikaRT::Error`](../Error.md) naming the input.

A graph that writes captured state in place (a traced function's closure KV cache or recurrence state) serves one stream of calls: the node runs its requests one at a time, and each advances that one state in the order they run. One such node shared by two pipelines meets the graph's one-run-at-a-time refusal (`FAILED_PRECONDITION`), and two graphs that capture the same state are not serialized against each other.

###  {#autotoc_md5_1s1}

#### KV-stateful graphs (`kv_cache_info()` non-empty) {#autotoc_md5}

The stateful arm serves graphs whose past planes are RUNTIME INPUTS (the compiled-model class): each session allocates one cache plane per past/present pair, shaped from the graph's own past-input specs with the batch and the sequence capacity taken from `kv` (pass `spec::KVSpecOptions(batch, capacity)`; such a graph refuses a zero capacity). A graph whose cache planes were captured as live tensors instead (surfaced present outputs with no past inputs) carries nothing a session plane could feed, so it serves through the stateless arm and its presents emit as ordinary outputs. An "ingest" phase stashes the request's data inputs; each "step" runs the graph with every past input fed AS A VIEW of its session plane (the valid prefix, zero-length before anything is cached) and lands every present output back into its plane, advancing the session's cache cursor. Cache planes never appear in the node's schema; past inputs and present outputs are session state, not request I/O. Scope: each session runs the graph once ("step" finishes after one pass); a generative feedback loop (next step's tokens from this step's outputs, a stop condition) is a decoder node's contract, not a graph adapter's; presents land in the planes through one device copy per step. A session's planes release with its state after the runtime evicts the session.

Header-only and caller-side: everything here, including any raise, compiles into your translation unit, never the library.

## `ClikaRT/runtime/model.h` {#model-h}

```cpp
#include <ClikaRT/runtime/model.h>
```

[`ClikaRT::runtime::Model`](./Model.md): the base a consumer subclasses to author a node in a serving pipeline (a tokenizer, an autoregressive decoder, a detokenizer). It is the runtime analogue of `nn::Module`'s override-seam: you declare a `schema()` + `phases()`, then implement `Phase_RunOnce` (a `RunOnce` pass) and, for a stepped model, `make_state` / `Phase_Iterate` / `evict` (the `Iterate` loop). The runtime drives these; you never call them yourself.

Authoring a leaf node: class MyNode : public runtime::Model \{ const ModelSchema\& schema() const override \{ return schema\_; \} Span\<const PhaseSpec\> phases() const override \{ return phases\_; \} void Phase\_RunOnce(const PhaseSpec\&, PhaseContext\& ctx) override \{ ... \} \}; Override methods MAY raise [`ClikaRT::Error`](../Error.md) on failure; the runtime contains the throw at the boundary and fails the session cleanly (it never unwinds into the engine). A stateful model also subclasses `ModelState` (see [state.h](./index.md#state-h)).

## `ClikaRT/runtime/pipeline.h` {#pipeline-h}

```cpp
#include <ClikaRT/runtime/pipeline.h>
```

[`ClikaRT::runtime::Pipeline`](./Pipeline.md): an ensemble of `Model` nodes (each its own continuous-batched executor) wired into one serving graph: a request flows node→node (e.g. Tokenizer → Decoder → Detokenizer), the pipeline routes each node's outputs to the next by name. This is the primary serving entry: build it from `PipelineStep`s, then `enqueue` requests and `await` (or stream via `RequestCallbacks`). Opaque handle (move-only). You keep every `Model*` referenced by the steps alive for the pipeline's lifetime.

## `ClikaRT/runtime/schema.h` {#schema-h}

```cpp
#include <ClikaRT/runtime/schema.h>
```

A `runtime::Model`'s I/O contract (`ModelSchema` of `TensorSpec`s) and its phase declarations (`PhaseSpec` of a `PhaseKind`). The runtime reads these to validate requests, route tensors between pipeline nodes, and drive each phase.

## `ClikaRT/runtime/serving.h` {#serving-h}

```cpp
#include <ClikaRT/runtime/serving.h>
```

The request/response vocabulary and the executor/pipeline configuration a consumer drives the runtime with. All carry only public types.

## `ClikaRT/runtime/state.h` {#state-h}

```cpp
#include <ClikaRT/runtime/state.h>
```

Per-session state bases for a stateful `runtime::Model`. A model that carries per-request state (a KV cache, a generation cursor) subclasses `ModelState`; the runtime owns one instance per live session (created by `Model::make_state`, destroyed after `Model::evict`). A state that also wants to participate in continuous batching additionally subclasses `BatchableState` so the batcher can pack heterogeneous-length sessions by reading `occupancy()` / `capacity()`, without naming the concrete type.