Skip to main content

ClikaRT::tokenizer::EncodeOptions

struct

Header: ClikaRT/tokenizer/tokenizer.h

Per-call controls for the tensor-shaped encode entry points.

Data members​

add_special_tokens​

bool add_special_tokens = true

Wrap the produced ids with the model's special tokens (bos/eos/...).

Declared in ClikaRT/tokenizer/tokenizer.h, line 44

return_tensors​

bool return_tensors = false

Build the Encoded tensor columns in addition to ids.

Declared in ClikaRT/tokenizer/tokenizer.h, line 46

varlen​

bool varlen = false

Lay input_ids out as a flat [ΣS] ragged buffer with a cu_seqlens offset table instead of a padded [B, S] matrix. Varlen never pads.

Declared in ClikaRT/tokenizer/tokenizer.h, line 49

pad_id​

std::optional<std::int32_t> pad_id = std::nullopt

Pad id for a padded batch; falls back to the tokenizer's default pad id.

Declared in ClikaRT/tokenizer/tokenizer.h, line 51

device​

StreamOrDevice device = {}

Placement for the produced tensors: a Device, or a Stream so the tensors land stream-ordered on that stream (ready to feed a forward on the same stream with no cross-stream wait). Unspecified falls back to the tokenizer's default placement (CPU unless configured otherwise).

Declared in ClikaRT/tokenizer/tokenizer.h, line 56

max_length​

std::optional<std::int64_t> max_length = std::nullopt

Cap each sequence at this many tokens, truncating per truncation_side. The model's special tokens are never cut: the core text is trimmed to (max_length − the specials' count) BEFORE bos/eos attach, so the total stays ≤ max_length. Unset = no truncation.

Declared in ClikaRT/tokenizer/tokenizer.h, line 61

truncation_side​

TruncationSide truncation_side = TruncationSide::Right

which end is trimmed when max_length binds

Declared in ClikaRT/tokenizer/tokenizer.h, line 62

padding_side​

PaddingSide padding_side = PaddingSide::Right

Which end a ragged padded batch fills. seq_lengths / cu_seqlens always carry the REAL lengths either way.

Declared in ClikaRT/tokenizer/tokenizer.h, line 65

pad_to_multiple_of​

std::optional<std::int64_t> pad_to_multiple_of = std::nullopt

Round the padded row length S up to a multiple of this (tensor-core friendly shapes). Padded [B, S] layout only (an error with varlen).

Declared in ClikaRT/tokenizer/tokenizer.h, line 68