Skip to main content

ClikaRT::spec::KVSpecOptions

struct

Header: ClikaRT/graph/input_specs.h

Options for generate_kv_input_specs: the batch size and past-sequence length every detected KV-cache input is specced at. The defaults describe a prefill step: a zero-length past. past_len_* follow ShapeProfile min/opt/max semantics; set all three equal (or just construct with one value) for a fixed past length.

Member functions​

KVSpecOptions()​

KVSpecOptions() =default

Declared in ClikaRT/graph/input_specs.h, line 265

KVSpecOptions(int64_t, int64_t)​

explicit KVSpecOptions(std::int64_t batch_size, std::int64_t past_len = 0)

Spec every detected KV input at batch_size with a fixed past_len.

Declared in ClikaRT/graph/input_specs.h, line 267

Data members​

batch​

std::int64_t batch = 1

batch size every detected KV input is specced at.

Declared in ClikaRT/graph/input_specs.h, line 260

past_len_min​

std::int64_t past_len_min = 0

smallest past length (ShapeProfile semantics).

Declared in ClikaRT/graph/input_specs.h, line 261

past_len_opt​

std::int64_t past_len_opt = 0

the past length the graph specializes to.

Declared in ClikaRT/graph/input_specs.h, line 262

past_len_max​

std::int64_t past_len_max = 0

largest past length (ShapeProfile semantics).

Declared in ClikaRT/graph/input_specs.h, line 263