`static` [`Tokenizer`](./Tokenizer.md) **`from_sentencepiece`**(`std::string_view` `path`)
Load a SentencePiece tokenizer model (a `tokenizer.model` file). Bare: the pieces keep their own ids, nothing frames the sequence, and there is no `tokenizer_config.json` overlay (bos/eos default to -1, no chat template); use [`from_huggingface`](./Tokenizer.md#from_huggingface) on the model directory for the class layout and those.
`static` [`Tokenizer`](./Tokenizer.md) **`from_tekken`**(`std::string_view` `path`)
Load a Mistral `tekken.json` tokenizer (tiktoken-based; carries its own special tokens). Bare, as above.
`static` [`Tokenizer`](./Tokenizer.md) **`from_gpt2`**(`std::string_view` `dir`)
Load a GPT-2-style byte-level BPE tokenizer from a directory containing `vocab.json` + `merges.txt`. Bare, as above.
`static` [`Tokenizer`](./Tokenizer.md) **`from_tiktoken`**(`std::string_view` `path`)
Load an OpenAI `.tiktoken` ranks file (one `base64(token_bytes) rank` line per token, e.g. whisper's `multilingual.tiktoken` / `gpt2.tiktoken`). The file carries no special tokens, so the loader synthesizes the OpenAI special-token set of the family its ranks-table size names:
- 50257 ranks, whisper MULTILINGUAL: the current 100-language set (what every large-v3-era checkpoint uses). A pre-large-v3 multilingual checkpoint uses a 99-language id layout the file alone cannot express (ids from `<|translate|>` on shift by one); assemble those with [`TokenizerBuilder`](./TokenizerBuilder.md).
- 50256 ranks, whisper ENGLISH-ONLY (the GPT-2 table): the 99-language set. For a plain GPT-2 tokenizer WITHOUT whisper specials, use [`from_gpt2`](./Tokenizer.md#from_gpt2) or [`TokenizerBuilder`](./TokenizerBuilder.md) instead. Any other table size raises [`ClikaRT::Error`](../Error.md) naming the size and the known families. Bare, as above. (See also [`from_tekken`](./Tokenizer.md#from_tekken), the other tiktoken-based format, whose JSON carries its own specials.)
`static` [`Tokenizer`](./Tokenizer.md) **`from_bert_vocab`**(`std::string_view` `dir`)
Load a BERT WordPiece tokenizer from a directory containing a bare `vocab.txt` (one token per line; id = line number), folding in the sibling `tokenizer_config.json` / `special_tokens_map.json` when present (the special-tokens map wins where both name a special). Bare, as above.
`static` [`Tokenizer`](./Tokenizer.md) **`from_file`**(`std::string_view` `path`)
Load from any supported tokenizer artifact, detecting the format for you; use this when you hold "the tokenizer file that shipped with the checkpoint" without knowing which format it is. A DIRECTORY probes, in order: `tokenizer.json` → `tokenizer.model` → `tekken.json` → `vocab.json` + `merges.txt` → `*.tiktoken` (lexicographically first), the same preference [`from_huggingface`](./Tokenizer.md#from_huggingface) uses, extended by `.tiktoken`. A FILE resolves by its canonical name/extension first; a nonstandard name is classified by content: JSON with a `model` object → `tokenizer.json`; JSON with a `vocab` array of `token_bytes` entries → `tekken.json`; `base64 rank` lines → `.tiktoken`; a SentencePiece model by its protobuf shape. A flat token→id JSON map is GPT-2's `vocab.json`, loadable only as the canonical `vocab.json` + `merges.txt` pair (pass the directory or the `vocab.json` itself). Each detected format then loads exactly as its named loader would: `tokenizer.json` and a SentencePiece model through [`from_huggingface`](./Tokenizer.md#from_huggingface) (including the `tokenizer_config.json` overlay; for a SentencePiece model that is the layout its declared `tokenizer_class` selects; with no config it loads as [`from_sentencepiece`](./Tokenizer.md#from_sentencepiece) loads it), the rest through their bare loaders above. An unrecognized path raises [`ClikaRT::Error`](../Error.md) naming every format tried.
[`Tokenizer`](./Tokenizer.md)`&` **`operator=`**([`Tokenizer`](./Tokenizer.md)`&&`) `noexcept`
Move-assign: transfers the tokenizer.
` binds to it directly.
Declared in `ClikaRT/tokenizer/tokenizer.h`, line 294
### tokenize() {#tokenize}
`std::vector<`[`Token`](./Token.md)`>` **`tokenize`**(`std::string_view` `text`, `bool` `add_special_tokens` `=` `true`) `const`
Encode `text` into per-token records: each [`Token`](./Token.md) carries its id, surface string, byte span `[begin, end)`, and whether it is a special token. Use this when you need token-to-source alignment (offsets) or the surface strings, not just the ids. `text` is read during the call, not retained.
Declared in `ClikaRT/tokenizer/tokenizer.h`, line 302
### encode\_text() {#encode_text}
[`Encoded`](./Encoded.md) **`encode_text`**(`std::string_view` `text`, `const` [`EncodeOptions`](./EncodeOptions.md)`&` `options` `=` `{}`) `const`
Encode one text into tensor form (a one-element batch). `ids` always carries the row; the [`Encoded`](./Encoded.md) tensor columns are filled per `options`.
Declared in `ClikaRT/tokenizer/tokenizer.h`, line 310
### encode\_batch() {#encode_batch}
[`Encoded`](./Encoded.md) **`encode_batch`**(
[`ClikaRT::Span`](../Span.md)`` `texts`,
`const` [`EncodeOptions`](./EncodeOptions.md)`&` `options` `=` `{}`
) `const`
Encode a batch of texts into tensor form: a padded `[B, S]` (default) or a flat varlen `[ΣS]` + `cu_seqlens` layout per `options`. `texts` is read during the call, not retained.
Declared in `ClikaRT/tokenizer/tokenizer.h`, line 316
### decode\_batch(Span\< Span\< int32\_t\>\>, bool) {#decode_batch}
`std::vector` **`decode_batch`**(
[`ClikaRT::Span`](../Span.md)`>` `sequences`,
`bool` `skip_special_tokens` `=` `true`
) `const`
Decode many id sequences (in parallel), one string per sequence, in order.
Declared in `ClikaRT/tokenizer/tokenizer.h`, line 324
### decode\_batch(Encoded, bool) {#decode_batch-2}
`std::vector` **`decode_batch`**(`const` [`Encoded`](./Encoded.md)`&` `encoded`, `bool` `skip_special_tokens` `=` `true`) `const`
Decode many id sequences (in parallel), one string per sequence, in order.
Declared in `ClikaRT/tokenizer/tokenizer.h`, line 331
### decode\_batch(Tensor, bool) {#decode_batch-3}
`std::vector` **`decode_batch`**(`const` [`Tensor`](../Tensor.md)`&` `ids`, `bool` `skip_special_tokens` `=` `true`) `const`
Decode many id sequences (in parallel), one string per sequence, in order.
Declared in `ClikaRT/tokenizer/tokenizer.h`, line 339
### decode\_batch(Tensor, Tensor, bool) {#decode_batch-4}
`std::vector` **`decode_batch`**(
`const` [`Tensor`](../Tensor.md)`&` `ids`,
`const` [`Tensor`](../Tensor.md)`&` `cu_seqlens`,
`bool` `skip_special_tokens` `=` `true`
) `const`
Decode many id sequences (in parallel), one string per sequence, in order.
Declared in `ClikaRT/tokenizer/tokenizer.h`, line 345
### bos\_id() {#bos_id}
`std::int64_t` **`bos_id`**() `const noexcept`
The begin-of-sequence / end-of-sequence / padding token id named by the model's config, or -1 when the config does not name one.
Declared in `ClikaRT/tokenizer/tokenizer.h`, line 350
### eos\_id() {#eos_id}
`std::int64_t` **`eos_id`**() `const noexcept`
the eos id; -1 when the config names none (shared block above)
Declared in `ClikaRT/tokenizer/tokenizer.h`, line 351
### pad\_id() {#pad_id}
`std::int64_t` **`pad_id`**() `const noexcept`
the pad id; -1 when the config names none (shared block above)
Declared in `ClikaRT/tokenizer/tokenizer.h`, line 352
### vocab\_size() {#vocab_size}
`std::int64_t` **`vocab_size`**() `const noexcept`
Number of entries in the model vocabulary.
Declared in `ClikaRT/tokenizer/tokenizer.h`, line 357
### token\_to\_id() {#token_to_id}
`std::int32_t` **`token_to_id`**(`std::string_view` `token`) `const`
The id of a surface token, or NotFound if the token is not in the vocabulary. `token` is read during the call, not retained.
Declared in `ClikaRT/tokenizer/tokenizer.h`, line 362
### id\_to\_token() {#id_to_token}
`std::string` **`id_to_token`**(`std::int32_t` `id`) `const`
The surface string for a token id, or NotFound if `id` is out of range.
Declared in `ClikaRT/tokenizer/tokenizer.h`, line 366
### has\_chat\_template() {#has_chat_template}
`bool` **`has_chat_template`**() `const noexcept`
True if the loaded tokenizer carries a chat template. [`from_huggingface`](./Tokenizer.md#from_huggingface) reads it beside a `tokenizer_config.json`, from the first of three sources that holds one: that file's `chat_template` key, a sibling `chat_template.jinja`, then a sibling `chat_template.json`.
Declared in `ClikaRT/tokenizer/tokenizer.h`, line 372
### apply\_chat\_template(json::Json, bool) {#apply_chat_template}
`std::string` **`apply_chat_template`**(`const` [`json::Json`](../json/Json.md)`&` `messages`, `bool` `add_generation_prompt` `=` `true`) `const`
Render chat `messages` into a prompt string using the model's chat template. `messages` is a JSON array of `{"role": ..., "content": ...}` objects; `add_generation_prompt` appends the assistant-turn priming so the model continues as the assistant. A failure (no chat template, malformed messages, or a render error) raises [`ClikaRT::Error`](../Error.md). `messages` is read during the call, not retained.
Declared in `ClikaRT/tokenizer/tokenizer.h`, line 381
### apply\_chat\_template(ChatTemplateInputs) {#apply_chat_template-2}
`std::string` **`apply_chat_template`**(`const` [`ChatTemplateInputs`](./ChatTemplateInputs.md)`&` `inputs`) `const`
Render chat `messages` into a prompt string using the model's chat template. `messages` is a JSON array of `{"role": ..., "content": ...}` objects; `add_generation_prompt` appends the assistant-turn priming so the model continues as the assistant. A failure (no chat template, malformed messages, or a render error) raises [`ClikaRT::Error`](../Error.md). `messages` is read during the call, not retained.
Declared in `ClikaRT/tokenizer/tokenizer.h`, line 390
### apply\_chat\_template(ChatTemplateInputs, ChatTemplateOptions) {#apply_chat_template-3}
`std::string` **`apply_chat_template`**(
`const` [`ChatTemplateInputs`](./ChatTemplateInputs.md)`&` `inputs`,
`const` [`ChatTemplateOptions`](./ChatTemplateOptions.md)`&` `options`
) `const`
Render chat `messages` into a prompt string using the model's chat template. `messages` is a JSON array of `{"role": ..., "content": ...}` objects; `add_generation_prompt` appends the assistant-turn priming so the model continues as the assistant. A failure (no chat template, malformed messages, or a render error) raises [`ClikaRT::Error`](../Error.md). `messages` is read during the call, not retained.
Declared in `ClikaRT/tokenizer/tokenizer.h`, line 396
### encode\_chat(json::Json, bool, bool) {#encode_chat}
`std::vector` **`encode_chat`**(
`const` [`json::Json`](../json/Json.md)`&` `messages`,
`bool` `add_generation_prompt` `=` `true`,
`bool` `add_special_tokens` `=` `false`
) `const`
Render chat `messages` with the model's chat template AND encode the result to token ids in one step, the messages→prompt→ids path. Encodes the rendered prompt with `add_special_tokens = false` by default: the template already writes its own bos/eos framing, so the tokenizer's post-processor must not add a second bos (the same law transformers' [`apply_chat_template`](./Tokenizer.md#apply_chat_template)`(`[`tokenize`](./Tokenizer.md#tokenize)`=True)` follows). Pass `true` only for a template that writes no bos itself; it additionally runs the post-processor framing, which is not what transformers does. `messages` is a JSON array of `{"role": ..., "content": ...}` objects, read during the call. A failure (no chat template, malformed messages, render error) raises [`ClikaRT::Error`](../Error.md).
Declared in `ClikaRT/tokenizer/tokenizer.h`, line 410
### encode\_chat(ChatTemplateInputs, ChatTemplateOptions, bool) {#encode_chat-2}
`std::vector` **`encode_chat`**(
`const` [`ChatTemplateInputs`](./ChatTemplateInputs.md)`&` `inputs`,
`const` [`ChatTemplateOptions`](./ChatTemplateOptions.md)`&` `options`,
`bool` `add_special_tokens` `=` `false`
) `const`
Render chat `messages` with the model's chat template AND encode the result to token ids in one step, the messages→prompt→ids path. Encodes the rendered prompt with `add_special_tokens = false` by default: the template already writes its own bos/eos framing, so the tokenizer's post-processor must not add a second bos (the same law transformers' [`apply_chat_template`](./Tokenizer.md#apply_chat_template)`(`[`tokenize`](./Tokenizer.md#tokenize)`=True)` follows). Pass `true` only for a template that writes no bos itself; it additionally runs the post-processor framing, which is not what transformers does. `messages` is a JSON array of `{"role": ..., "content": ...}` objects, read during the call. A failure (no chat template, malformed messages, render error) raises [`ClikaRT::Error`](../Error.md).
Declared in `ClikaRT/tokenizer/tokenizer.h`, line 423
### streaming\_decoder() {#streaming_decoder}
[`StreamingDecoder`](./StreamingDecoder.md) **`streaming_decoder`**(`bool` `skip_special_tokens` `=` `true`) `const`
A streaming id→text decoder sharing this tokenizer's internals, the handle stays valid even after this [`Tokenizer`](./Tokenizer.md) is destroyed. One decoder serves one generation stream at a time (`finish` resets it). Infallible.
Declared in `ClikaRT/tokenizer/tokenizer.h`, line 430