ClikaRT::tokenizer::Tokenizer
class
Header: ClikaRT/tokenizer/tokenizer.h
A loaded tokenizer. Move-only.
Every fallible operation returns its value directly and raises ClikaRT::Error on failure. Wrap a call in CLIKART_TRY(...) if you would rather inspect a Result than catch.
Static member functions
from_huggingface()
static Tokenizer from_huggingface(std::string_view path)
Load from a HuggingFace tokenizer: path is a model directory OR a single artifact file (tokenizer.json / tokenizer.model / sentencepiece.bpe.model / tekken.json / vocab.json+merges.txt). Auto-detects the artifact and overlays a sibling tokenizer_config.json (bos/eos ids + chat template) when present.
A bare SentencePiece artifact (tokenizer.model / sentencepiece.bpe.model) beside a tokenizer_config.json naming its tokenizer_class is laid out the way that class's tokenizer would be: XLM-RoBERTa / LayoutXLM, CamemBERT, BARThez, NLLB, mBART and XGLM get their reserved head ids (<s> 0, <pad> 1, </s> 2, <unk> 3), their language codes, <mask>, and their <s> … </s> framing (NLLB and mBART frame with the config's src_lang, or the class default); the Llama family (Llama, CodeLlama, Phi-3) gets <s> / </s> framing per add_bos_token / add_eos_token; any other class, or no config, loads the pieces with their own ids and no framing.
Declared in ClikaRT/tokenizer/tokenizer.h, line 209
from_sentencepiece()
static Tokenizer from_sentencepiece(std::string_view path)
Load a SentencePiece tokenizer model (a tokenizer.model file). Bare: the pieces keep their own ids, nothing frames the sequence, and there is no tokenizer_config.json overlay (bos/eos default to -1, no chat template); use from_huggingface on the model directory for the class layout and those.
Declared in ClikaRT/tokenizer/tokenizer.h, line 216
from_tekken()
static Tokenizer from_tekken(std::string_view path)
Load a Mistral tekken.json tokenizer (tiktoken-based; carries its own special tokens). Bare, as above.
Declared in ClikaRT/tokenizer/tokenizer.h, line 221
from_gpt2()
static Tokenizer from_gpt2(std::string_view dir)
Load a GPT-2-style byte-level BPE tokenizer from a directory containing vocab.json + merges.txt. Bare, as above.
Declared in ClikaRT/tokenizer/tokenizer.h, line 226
from_tiktoken()
static Tokenizer from_tiktoken(std::string_view path)
Load an OpenAI .tiktoken ranks file (one base64(token_bytes) rank line per token, e.g. whisper's multilingual.tiktoken / gpt2.tiktoken). The file carries no special tokens, so the loader synthesizes the OpenAI special-token set of the family its ranks-table size names:
- 50257 ranks, whisper MULTILINGUAL: the current 100-language set (what every large-v3-era checkpoint uses). A pre-large-v3 multilingual checkpoint uses a 99-language id layout the file alone cannot express (ids from
<|translate|>on shift by one); assemble those withTokenizerBuilder. - 50256 ranks, whisper ENGLISH-ONLY (the GPT-2 table): the 99-language set. For a plain GPT-2 tokenizer WITHOUT whisper specials, use
from_gpt2orTokenizerBuilderinstead. Any other table size raisesClikaRT::Errornaming the size and the known families. Bare, as above. (See alsofrom_tekken, the other tiktoken-based format, whose JSON carries its own specials.)
Declared in ClikaRT/tokenizer/tokenizer.h, line 244
from_bert_vocab()
static Tokenizer from_bert_vocab(std::string_view dir)
Load a BERT WordPiece tokenizer from a directory containing a bare vocab.txt (one token per line; id = line number), folding in the sibling tokenizer_config.json / special_tokens_map.json when present (the special-tokens map wins where both name a special). Bare, as above.
Declared in ClikaRT/tokenizer/tokenizer.h, line 252
from_file()
static Tokenizer from_file(std::string_view path)
Load from any supported tokenizer artifact, detecting the format for you; use this when you hold "the tokenizer file that shipped with the checkpoint" without knowing which format it is. A DIRECTORY probes, in order: tokenizer.json → tokenizer.model → tekken.json → vocab.json + merges.txt → *.tiktoken (lexicographically first), the same preference from_huggingface uses, extended by .tiktoken. A FILE resolves by its canonical name/extension first; a nonstandard name is classified by content: JSON with a model object → tokenizer.json; JSON with a vocab array of token_bytes entries → tekken.json; base64 rank lines → .tiktoken; a SentencePiece model by its protobuf shape. A flat token→id JSON map is GPT-2's vocab.json, loadable only as the canonical vocab.json + merges.txt pair (pass the directory or the vocab.json itself). Each detected format then loads exactly as its named loader would: tokenizer.json and a SentencePiece model through from_huggingface (including the tokenizer_config.json overlay; for a SentencePiece model that is the layout its declared tokenizer_class selects; with no config it loads as from_sentencepiece loads it), the rest through their bare loaders above. An unrecognized path raises ClikaRT::Error naming every format tried.
Declared in ClikaRT/tokenizer/tokenizer.h, line 274
Member functions
Tokenizer()
Tokenizer(Tokenizer&&) noexcept
Declared in ClikaRT/tokenizer/tokenizer.h, line 276
operator=()
Move-assign: transfers the tokenizer.
Declared in ClikaRT/tokenizer/tokenizer.h, line 278
~Tokenizer()
~Tokenizer()
Releases the tokenizer (outstanding StreamingDecoders stay valid).
Declared in ClikaRT/tokenizer/tokenizer.h, line 280
encode()
std::vector<std::int32_t> encode(std::string_view text, bool add_special_tokens = true) const
Encode text into token ids. text is read during the call (not retained).
Declared in ClikaRT/tokenizer/tokenizer.h, line 284
decode()
std::string decode(ClikaRT::Span<const std::int32_t> ids, bool skip_special_tokens = true) const
Decode token ids back into text. ids is a non-owning view read during the call; a std::vector<std::int32_t> binds to it directly.
Declared in ClikaRT/tokenizer/tokenizer.h, line 290
tokenize()
std::vector<Token> tokenize(std::string_view text, bool add_special_tokens = true) const
Encode text into per-token records: each Token carries its id, surface string, byte span [begin, end), and whether it is a special token. Use this when you need token-to-source alignment (offsets) or the surface strings, not just the ids. text is read during the call, not retained.
Declared in ClikaRT/tokenizer/tokenizer.h, line 298
encode_text()
Encoded encode_text(std::string_view text, const EncodeOptions& options = {}) const
Encode one text into tensor form (a one-element batch). ids always carries the row; the Encoded tensor columns are filled per options.
Declared in ClikaRT/tokenizer/tokenizer.h, line 306
encode_batch()
Encoded encode_batch(
ClikaRT::Span<const std::string_view> texts,
const EncodeOptions& options = {}
) const
Encode a batch of texts into tensor form: a padded [B, S] (default) or a flat varlen [ΣS] + cu_seqlens layout per options. texts is read during the call, not retained.
Declared in ClikaRT/tokenizer/tokenizer.h, line 312
decode_batch(Span< Span< int32_t>>, bool)
std::vector<std::string> decode_batch(
ClikaRT::Span<const ClikaRT::Span<const std::int32_t>> sequences,
bool skip_special_tokens = true
) const
Decode many id sequences (in parallel), one string per sequence, in order.
Declared in ClikaRT/tokenizer/tokenizer.h, line 320
decode_batch(Encoded, bool)
std::vector<std::string> decode_batch(const Encoded& encoded, bool skip_special_tokens = true) const
Decode many id sequences (in parallel), one string per sequence, in order.
Declared in ClikaRT/tokenizer/tokenizer.h, line 327
decode_batch(Tensor, bool)
std::vector<std::string> decode_batch(const Tensor& ids, bool skip_special_tokens = true) const
Decode many id sequences (in parallel), one string per sequence, in order.
Declared in ClikaRT/tokenizer/tokenizer.h, line 335
decode_batch(Tensor, Tensor, bool)
std::vector<std::string> decode_batch(
const Tensor& ids,
const Tensor& cu_seqlens,
bool skip_special_tokens = true
) const
Decode many id sequences (in parallel), one string per sequence, in order.
Declared in ClikaRT/tokenizer/tokenizer.h, line 341
bos_id()
std::int64_t bos_id() const noexcept
The begin-of-sequence / end-of-sequence / padding token id named by the model's config, or -1 when the config does not name one.
Declared in ClikaRT/tokenizer/tokenizer.h, line 346
eos_id()
std::int64_t eos_id() const noexcept
the eos id; -1 when the config names none (shared block above)
Declared in ClikaRT/tokenizer/tokenizer.h, line 347
pad_id()
std::int64_t pad_id() const noexcept
the pad id; -1 when the config names none (shared block above)
Declared in ClikaRT/tokenizer/tokenizer.h, line 348
vocab_size()
std::int64_t vocab_size() const noexcept
Number of entries in the model vocabulary.
Declared in ClikaRT/tokenizer/tokenizer.h, line 353
token_to_id()
std::int32_t token_to_id(std::string_view token) const
The id of a surface token, or NotFound if the token is not in the vocabulary. token is read during the call, not retained.
Declared in ClikaRT/tokenizer/tokenizer.h, line 358
id_to_token()
std::string id_to_token(std::int32_t id) const
The surface string for a token id, or NotFound if id is out of range.
Declared in ClikaRT/tokenizer/tokenizer.h, line 362
has_chat_template()
bool has_chat_template() const noexcept
True if the loaded tokenizer carries a chat template (from tokenizer_config.json).
Declared in ClikaRT/tokenizer/tokenizer.h, line 366
apply_chat_template(json::Json, bool)
std::string apply_chat_template(const json::Json& messages, bool add_generation_prompt = true) const
Render chat messages into a prompt string using the model's chat template. messages is a JSON array of {"role": ..., "content": ...} objects; add_generation_prompt appends the assistant-turn priming so the model continues as the assistant. A failure (no chat template, malformed messages, or a render error) raises ClikaRT::Error. messages is read during the call, not retained.
Declared in ClikaRT/tokenizer/tokenizer.h, line 375
apply_chat_template(ChatTemplateInputs)
std::string apply_chat_template(const ChatTemplateInputs& inputs) const
Render chat messages into a prompt string using the model's chat template. messages is a JSON array of {"role": ..., "content": ...} objects; add_generation_prompt appends the assistant-turn priming so the model continues as the assistant. A failure (no chat template, malformed messages, or a render error) raises ClikaRT::Error. messages is read during the call, not retained.
Declared in ClikaRT/tokenizer/tokenizer.h, line 384
apply_chat_template(ChatTemplateInputs, ChatTemplateOptions)
std::string apply_chat_template(
const ChatTemplateInputs& inputs,
const ChatTemplateOptions& options
) const
Render chat messages into a prompt string using the model's chat template. messages is a JSON array of {"role": ..., "content": ...} objects; add_generation_prompt appends the assistant-turn priming so the model continues as the assistant. A failure (no chat template, malformed messages, or a render error) raises ClikaRT::Error. messages is read during the call, not retained.
Declared in ClikaRT/tokenizer/tokenizer.h, line 390
encode_chat(json::Json, bool, bool)
std::vector<std::int32_t> encode_chat(
const json::Json& messages,
bool add_generation_prompt = true,
bool add_special_tokens = false
) const
Render chat messages with the model's chat template AND encode the result to token ids in one step, the messages→prompt→ids path. Encodes the rendered prompt with add_special_tokens = false by default: the template already writes its own bos/eos framing, so the tokenizer's post-processor must not add a second bos (the same law transformers' apply_chat_template(tokenize=True) follows). Pass true only for a template that writes no bos itself; it additionally runs the post-processor framing, which is not what transformers does. messages is a JSON array of {"role": ..., "content": ...} objects, read during the call. A failure (no chat template, malformed messages, render error) raises ClikaRT::Error.
Declared in ClikaRT/tokenizer/tokenizer.h, line 404
encode_chat(ChatTemplateInputs, ChatTemplateOptions, bool)
std::vector<std::int32_t> encode_chat(
const ChatTemplateInputs& inputs,
const ChatTemplateOptions& options,
bool add_special_tokens = false
) const
Render chat messages with the model's chat template AND encode the result to token ids in one step, the messages→prompt→ids path. Encodes the rendered prompt with add_special_tokens = false by default: the template already writes its own bos/eos framing, so the tokenizer's post-processor must not add a second bos (the same law transformers' apply_chat_template(tokenize=True) follows). Pass true only for a template that writes no bos itself; it additionally runs the post-processor framing, which is not what transformers does. messages is a JSON array of {"role": ..., "content": ...} objects, read during the call. A failure (no chat template, malformed messages, render error) raises ClikaRT::Error.
Declared in ClikaRT/tokenizer/tokenizer.h, line 417
streaming_decoder()
StreamingDecoder streaming_decoder(bool skip_special_tokens = true) const
A streaming id→text decoder sharing this tokenizer's internals, the handle stays valid even after this Tokenizer is destroyed. One decoder serves one generation stream at a time (finish resets it). Infallible.
Declared in ClikaRT/tokenizer/tokenizer.h, line 424