ClikaRT::tokenizer

namespace

Classes

Name	Description
BpeOptions	Extra BPE knobs; every field is optional and defaults to HF's behavior.
ChatTemplateInputs	The full per-render inputs for apply_chat_template: the chat turns plus everything a tool-calling or context-aware template consumes. All JSON values are read during the call, not retained.
ChatTemplateOptions	Per-call overrides for apply_chat_template / encode_chat. Every toggle is tri-state: unset keeps the template's standing behavior, set forces the value for THIS render only. chat_template swaps the template SOURCE for the call; it is parsed and capability-probed per render (keep the standing template for the hot path) and requires the tokenizer to carry a template, whose bos/eos binding the override inherits.
Decoder	The detokenization stage (optional): turns token strings back into text (reverse byte-level / metaspace / WordPiece, fuse, strip). Composable via sequence(); move-only handle consumed by TokenizerBuilder::decoder.
Encoded	Tensor-shaped encode output. ids always carries the per-sequence id rows (one inner vector per input text). The tensor columns are present only when EncodeOptions::return_tensors is set:input_ids: [B, S] padded Int32 (per EncodeOptions::padding_side, right by default) or [ΣS] flat Int32 (varlen). attention_mask: [B, S] Int32, 1 for a token / 0 for pad (padded path only). seq_lengths: [B] Int32, the per-sequence token counts. cu_seqlens: [B + 1] Int32, the exclusive prefix-sum offset table, the varlen attention contract's index dtype, so it feeds attention directly.
EncodeOptions	Per-call controls for the tensor-shaped encode entry points.
Model	The core tokenization stage (required): BPE / Unigram / WordLevel / WordPiece, built from an explicit vocabulary; no file needed. Consumed by TokenizerBuilder's constructor; move-only handle.
Normalizer	The text-cleanup stage (optional): Unicode normalization, case folding, stripping, regex replacement. Composable via sequence(); each factory returns a move-only handle TokenizerBuilder::normalizer consumes.
PostProcessor	The special-token / template stage (optional): wraps encoded sequences with special tokens ([CLS] A [SEP]-style templates, byte-level offset re-encoding). Move-only handle consumed by TokenizerBuilder::post_processor.
PreTokenizer	The pre-segmentation stage (optional): cuts text into pieces the model tokenizes independently (whitespace, byte-level, metaspace, regex splits). Composable via sequence(); move-only handle consumed by TokenizerBuilder::pre_tokenizer.
StreamingDecoder	Incremental id→text decoding for a generation stream. Ids arrive one at a time, but decoded bytes are only displayable at UTF-8 boundaries (a code point can span tokens); push returns exactly the newly-stable text (often empty while a code point is incomplete) and finish flushes the tail and resets the decoder for a fresh stream. Obtain one from Tokenizer::streaming_decoder; the handle shares the tokenizer's internals and stays valid even after that Tokenizer is destroyed. Move-only.
Template	A parsed, reusable template. Move-only.
Token	One token produced by tokenize: its id, the surface string it spells, and the half-open byte span [begin, end) it covers in the source text. special is true for a token the tokenizer injected itself (bos/eos/...), not one that came from the input.
Tokenizer	A loaded tokenizer. Move-only.
TokenizerBuilder	Assembles a Tokenizer from a required Model plus the optional pipeline stages.

Enumerations

enum TruncationSide

enum class TruncationSide : std::uint8_t

Which end of a too-long sequence truncation drops (Right keeps the head).

Enumerator	Value	Description
Right	0
Left

Declared in ClikaRT/tokenizer/tokenizer.h, line 35

enum PaddingSide

enum class PaddingSide : std::uint8_t

Which end of a short row a padded batch fills. Left suits decoder-only batch generation; every row's real tokens end flush at the last column.

Enumerator	Value	Description
Right	0
Left

Declared in ClikaRT/tokenizer/tokenizer.h, line 39

Type aliases

using VocabEntry

using VocabEntry = std::pair<std::string_view, std::int32_t>

A (token, id) vocabulary entry. The views are non-owning; keep their backing alive for the factory call that consumes them.

Declared in ClikaRT/tokenizer/builder.h, line 41

using MergeRule

using MergeRule = std::pair<std::string_view, std::string_view>

A BPE merge rule (left, right), ranked by position in the list.

Declared in ClikaRT/tokenizer/builder.h, line 43

using UnigramEntry

using UnigramEntry = std::pair<std::string_view, double>

A Unigram vocabulary entry (token, log-probability score).

Declared in ClikaRT/tokenizer/builder.h, line 45

ClikaRT/tokenizer/builder.h

#include <ClikaRT/tokenizer/builder.h>

Build a Tokenizer from scratch (a model plus an optional normalizer / pre-tokenizer / post-processor / decoder pipeline) without a tokenizer file. Each pipeline stage is an opaque handle produced by a named factory (Normalizer::nfc(), Model::bpe(...), Decoder::byte_level(), …); TokenizerBuilder assembles them into a Tokenizer.

A factory that can fail (a bad regex, a merge that names a missing token) raises ClikaRT::Error on failure, like the rest of the public surface; the infallible ones return the handle directly. The handles are move-only; a builder consumes each stage you hand it.