Skip to main content

ClikaRT::runtime::BatchableState

class

Header: ClikaRT/runtime/state.h

Mix-in for a ModelState that the continuous batcher can pack. The batcher reads these to decide how many sessions fit in one step; both must be host-cheap (no device round-trip). The getters are noexcept: an override that raises terminates in your own program rather than unwinding into the runtime, and the compiler refuses an override that is not noexcept.

Member functions​

~BatchableState()​

virtual ~BatchableState()

Virtual; deletable through the interface.

Declared in ClikaRT/runtime/state.h, line 46

occupancy()​

virtual std::int64_t occupancy() const noexcept=0

Current per-session load (e.g. cached tokens) and the per-session ceiling (e.g. the KV-cache capacity). The batcher uses the headroom (capacity-occupancy) to admit work.

Declared in ClikaRT/runtime/state.h, line 51

capacity()​

virtual std::int64_t capacity() const noexcept=0

Total session capacity of this state (occupancy() is the live count).

Declared in ClikaRT/runtime/state.h, line 53

step_tokens()​

virtual std::int64_t step_tokens() const noexcept

Units this session wants to RUN in its next step, the per-step cost, where occupancy() is the carried state. A decode step is 1; a session ingesting a prompt in chunks reports its next chunk's length. The batcher sums it against BatcherConfig::step_token_budget so one step's launched work stays bounded. Host-cheap. Default: one unit.

Declared in ClikaRT/runtime/state.h, line 60