---
title: "ClikaRT::tokenizer"
sidebar_label: "tokenizer"
description: "Namespace ClikaRT::tokenizer"
---

<!-- Generated by tools/doxygen2mdx from Doxygen XML. Do not edit. -->

<span class="api-kind api-kind--namespace">namespace</span>

## Classes

| Name | Description |
| --- | --- |
| [`BpeOptions`](./BpeOptions.md) | Extra BPE knobs; every field is optional and defaults to HF's behavior. |
| [`ChatTemplateInputs`](./ChatTemplateInputs.md) | The full per-render inputs for `apply_chat_template`: the chat turns plus everything a tool-calling or context-aware template consumes. All JSON values are read during the call, not retained. |
| [`ChatTemplateOptions`](./ChatTemplateOptions.md) | Per-call overrides for `apply_chat_template` / `encode_chat`. Every toggle is tri-state: unset keeps the template's standing behavior, set forces the value for THIS render only. [`chat_template`](./ChatTemplateOptions.md#chat_template) swaps the template SOURCE for the call; it is parsed and capability-probed per render (keep the standing template for the hot path) and requires the tokenizer to carry a template, whose bos/eos binding the override inherits. |
| [`Decoder`](./Decoder.md) | The detokenization stage (optional): turns token strings back into text (reverse byte-level / metaspace / WordPiece, fuse, strip). Composable via [`sequence()`](./Decoder.md#sequence); move-only handle consumed by [`TokenizerBuilder::decoder`](./TokenizerBuilder.md#decoder). |
| [`Encoded`](./Encoded.md) | Tensor-shaped encode output. [`ids`](./Encoded.md#ids) always carries the per-sequence id rows (one inner vector per input text). The tensor columns are present only when [`EncodeOptions::return_tensors`](./EncodeOptions.md#return_tensors) is set:input\_ids: \[B, S\] padded Int32 (per EncodeOptions::padding\_side, right by default) or \[ΣS\] flat Int32 (varlen). attention\_mask: \[B, S\] Int32, 1 for a token / 0 for pad (padded path only). seq\_lengths: \[B\] Int32, the per-sequence token counts. cu\_seqlens: \[B + 1\] Int32, the exclusive prefix-sum offset table, the varlen attention contract's index dtype, so it feeds attention directly. |
| [`EncodeOptions`](./EncodeOptions.md) | Per-call controls for the tensor-shaped encode entry points. |
| [`Model`](./Model.md) | The core tokenization stage (required): BPE / Unigram / WordLevel / WordPiece, built from an explicit vocabulary; no file needed. Consumed by [`TokenizerBuilder`](./TokenizerBuilder.md)'s constructor; move-only handle. |
| [`Normalizer`](./Normalizer.md) | The text-cleanup stage (optional): Unicode normalization, case folding, stripping, regex replacement. Composable via [`sequence()`](./Normalizer.md#sequence); each factory returns a move-only handle [`TokenizerBuilder::normalizer`](./TokenizerBuilder.md#normalizer) consumes. |
| [`PostProcessor`](./PostProcessor.md) | The special-token / template stage (optional): wraps encoded sequences with special tokens (`[CLS] A [SEP]`-style templates, byte-level offset re-encoding). Move-only handle consumed by [`TokenizerBuilder::post_processor`](./TokenizerBuilder.md#post_processor). |
| [`PreTokenizer`](./PreTokenizer.md) | The pre-segmentation stage (optional): cuts text into pieces the model tokenizes independently (whitespace, byte-level, metaspace, regex splits). Composable via [`sequence()`](./PreTokenizer.md#sequence); move-only handle consumed by [`TokenizerBuilder::pre_tokenizer`](./TokenizerBuilder.md#pre_tokenizer). |
| [`StreamingDecoder`](./StreamingDecoder.md) | Incremental id→text decoding for a generation stream. Ids arrive one at a time, but decoded bytes are only displayable at UTF-8 boundaries (a code point can span tokens); [`push`](./StreamingDecoder.md#push) returns exactly the newly-stable text (often empty while a code point is incomplete) and [`finish`](./StreamingDecoder.md#finish) flushes the tail and resets the decoder for a fresh stream. Obtain one from [`Tokenizer::streaming_decoder`](./Tokenizer.md#streaming_decoder); the handle shares the tokenizer's internals and stays valid even after that [`Tokenizer`](./Tokenizer.md) is destroyed. Move-only. |
| [`Template`](./Template.md) | A parsed, reusable template. Move-only. |
| [`Token`](./Token.md) | One token produced by `tokenize`: its id, the surface string it spells, and the half-open byte span `[`[`begin`](./Token.md#begin)`, `[`end`](./Token.md#end)`)` it covers in the source text. [`special`](./Token.md#special) is true for a token the tokenizer injected itself (bos/eos/...), not one that came from the input. |
| [`Tokenizer`](./Tokenizer.md) | A loaded tokenizer. Move-only. |
| [`TokenizerBuilder`](./TokenizerBuilder.md) | Assembles a [`Tokenizer`](./Tokenizer.md) from a required [`Model`](./Model.md) plus the optional pipeline stages. |

## Enumerations

### enum TruncationSide {#TruncationSide}

<div class="api-signature">

`enum class` **`TruncationSide`** `:` `std::uint8_t`

</div>

Which end of a too-long sequence truncation drops (`Right` keeps the head).

| Enumerator | Value | Description |
| --- | --- | --- |
| `Right` | `0` |  |
| `Left` |  |  |

<div class="api-declared">

Declared in `ClikaRT/tokenizer/tokenizer.h`, line 35

</div>

### enum PaddingSide {#PaddingSide}

<div class="api-signature">

`enum class` **`PaddingSide`** `:` `std::uint8_t`

</div>

Which end of a short row a padded batch fills. `Left` suits decoder-only batch generation; every row's real tokens end flush at the last column.

| Enumerator | Value | Description |
| --- | --- | --- |
| `Right` | `0` |  |
| `Left` |  |  |

<div class="api-declared">

Declared in `ClikaRT/tokenizer/tokenizer.h`, line 39

</div>

## Type aliases

### using VocabEntry {#VocabEntry}

<div class="api-signature">

`using` **`VocabEntry`** `=` `std::pair<std::string_view, std::int32_t>`

</div>

A `(token, id)` vocabulary entry. The views are non-owning; keep their backing alive for the factory call that consumes them.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 41

</div>

### using MergeRule {#MergeRule}

<div class="api-signature">

`using` **`MergeRule`** `=` `std::pair<std::string_view, std::string_view>`

</div>

A BPE merge rule `(left, right)`, ranked by position in the list.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 43

</div>

### using UnigramEntry {#UnigramEntry}

<div class="api-signature">

`using` **`UnigramEntry`** `=` `std::pair<std::string_view, double>`

</div>

A Unigram vocabulary entry `(token, log-probability score)`.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 45

</div>

## `ClikaRT/tokenizer/builder.h` {#builder-h}

```cpp
#include <ClikaRT/tokenizer/builder.h>
```

Build a `Tokenizer` from scratch (a model plus an optional normalizer / pre-tokenizer / post-processor / decoder pipeline) without a tokenizer file. Each pipeline stage is an opaque handle produced by a named factory (`Normalizer::nfc()`, `Model::bpe(...)`, `Decoder::byte_level()`, …); `TokenizerBuilder` assembles them into a `Tokenizer`.

A factory that can fail (a bad regex, a merge that names a missing token) raises [`ClikaRT::Error`](../Error.md) on failure, like the rest of the public surface; the infallible ones return the handle directly. The handles are move-only; a builder consumes each stage you hand it.