ClikaRT::runtime::BatcherConfig

struct

Header: ClikaRT/runtime/serving.h

Continuous-batch composition knobs.

Data members

max_batch

std::size_t max_batch = 0

max sessions per step (0 = engine default)

Declared in ClikaRT/runtime/serving.h, line 77

work_budget

std::int64_t work_budget = 0

per-step work cap (0 = unbounded)

Declared in ClikaRT/runtime/serving.h, line 78

step_token_budget

std::int64_t step_token_budget = 0

Cap on the tokens LAUNCHED per step, summed across the admitted sessions (0 = unbounded). Bounds a step's latency under mixed prompt-ingestion + decode traffic: a session whose next-step tokens don't fit the remaining budget waits for a later step. The first admitted session is exempt, so progress is always made.

Declared in ClikaRT/runtime/serving.h, line 84