---
title: "ClikaRT::tokenizer::TokenizerBuilder"
sidebar_label: "TokenizerBuilder"
description: "Assembles a Tokenizer from a required Model plus the optional pipeline stages."
---

<!-- Generated by tools/doxygen2mdx from Doxygen XML. Do not edit. -->

<span class="api-kind api-kind--class">class</span>

**Header:** `ClikaRT/tokenizer/builder.h`

Assembles a [`Tokenizer`](./Tokenizer.md) from a required [`Model`](./Model.md) plus the optional pipeline stages.

Fluent: every setter consumes its stage handle and returns `*this`, so a build reads as one chain ending in [`build()`](./TokenizerBuilder.md#build), which validates the assembled configuration (template/special-token consistency, the chat template compiles) and raises [`ClikaRT::Error`](../Error.md) on an inconsistent one.

```cpp
auto tok = ClikaRT::TokenizerBuilder(std::move(model))
               .normalizer(ClikaRT::Normalizer::nfc())
               .pre_tokenizer(ClikaRT::PreTokenizer::byte_level())
               .eos_id(2)
               .build();
```

## Member functions

### TokenizerBuilder(Model, StreamOrDevice) {#TokenizerBuilder}

<div class="api-signature">

`explicit` **`TokenizerBuilder`**([`Model`](./Model.md) `model`, [`StreamOrDevice`](../StreamOrDevice.md) `where` `=` `{}`)

</div>

Start from the required model. `where` is the default placement for the tokenizer's tensor-shaped entry points.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 396

</div>

### normalizer() {#normalizer}

<div class="api-signature">

[`TokenizerBuilder`](./TokenizerBuilder.md)`&` **`normalizer`**([`Normalizer`](./Normalizer.md) `normalizer`)

</div>

Set a pipeline stage (each consumes its handle). Chainable.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 399

</div>

### pre\_tokenizer() {#pre_tokenizer}

<div class="api-signature">

[`TokenizerBuilder`](./TokenizerBuilder.md)`&` **`pre_tokenizer`**([`PreTokenizer`](./PreTokenizer.md) `pre_tokenizer`)

</div>

Set the pre-segmentation stage (consumes the handle). Chainable.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 402

</div>

### post\_processor() {#post_processor}

<div class="api-signature">

[`TokenizerBuilder`](./TokenizerBuilder.md)`&` **`post_processor`**([`PostProcessor`](./PostProcessor.md) `post_processor`)

</div>

Set the special-token / template stage (consumes the handle). Chainable.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 405

</div>

### decoder() {#decoder}

<div class="api-signature">

[`TokenizerBuilder`](./TokenizerBuilder.md)`&` **`decoder`**([`Decoder`](./Decoder.md) `decoder`)

</div>

Set the detokenization stage (consumes the handle). Chainable.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 408

</div>

### chat\_template() {#chat_template}

<div class="api-signature">

[`TokenizerBuilder`](./TokenizerBuilder.md)`&` **`chat_template`**(<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::string_view` `jinja_source`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::string_view` `bos_token`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::string_view` `eos_token`<br />
)

</div>

Attach a chat template (a Jinja source string) so [`Tokenizer`](./Tokenizer.md)'s `apply_chat_template` / `encode_chat` can render chat turns. `bos_token` / `eos_token` are the framing tokens the template references. The source is compiled at [`build()`](./TokenizerBuilder.md#build); an invalid template surfaces as a [`build()`](./TokenizerBuilder.md#build) error.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 414

</div>

### add\_token() {#add_token}

<div class="api-signature">

[`TokenizerBuilder`](./TokenizerBuilder.md)`&` **`add_token`**(<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::string_view` `content`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::int32_t` `id`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`bool` `single_word` `=` `false`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`bool` `lstrip` `=` `false`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`bool` `rstrip` `=` `false`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`bool` `normalized` `=` `true`<br />
)

</div>

Register an added token: an ordinary added token ([`add_token`](./TokenizerBuilder.md#add_token)) or a special one ([`add_special_token`](./TokenizerBuilder.md#add_special_token), maskable / skippable on decode). Register an ordinary added token.

**Parameters**

- `content`: the token text.
- `id`: its id.
- `single_word`: match only at word boundaries.
- `lstrip`: consume whitespace to the token's left.
- `rstrip`: consume whitespace to the token's right.
- `normalized`: match against normalized (vs raw) text.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 426

</div>

### add\_special\_token() {#add_special_token}

<div class="api-signature">

[`TokenizerBuilder`](./TokenizerBuilder.md)`&` **`add_special_token`**(`std::string_view` `content`, `std::int32_t` `id`)

</div>

Register the SPECIAL added token `content` under `id` (special tokens are maskable / skippable on decode).

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 431

</div>

### pad\_id() {#pad_id}

<div class="api-signature">

[`TokenizerBuilder`](./TokenizerBuilder.md)`&` **`pad_id`**(`std::int32_t` `id`)

</div>

Well-known token ids surfaced by the tokenizer (defaults: none / -1).

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 434

</div>

### bos\_id() {#bos_id}

<div class="api-signature">

[`TokenizerBuilder`](./TokenizerBuilder.md)`&` **`bos_id`**(`std::int32_t` `id`)

</div>

Set the beginning-of-sequence token id. Chainable.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 437

</div>

### eos\_id() {#eos_id}

<div class="api-signature">

[`TokenizerBuilder`](./TokenizerBuilder.md)`&` **`eos_id`**(`std::int32_t` `id`)

</div>

Set the end-of-sequence token id. Chainable.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 439

</div>

### build() {#build}

<div class="api-signature">

[`Tokenizer`](./Tokenizer.md) **`build`**()

</div>

Assemble the tokenizer; raises on an inconsistent configuration.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 443

</div>

### TokenizerBuilder(TokenizerBuilder) {#TokenizerBuilder-2}

<div class="api-signature">

**`TokenizerBuilder`**(`TokenizerBuilder&&` `other`) `noexcept`

</div>

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 445

</div>

### operator=() {#operator-assign}

<div class="api-signature">

[`TokenizerBuilder`](./TokenizerBuilder.md)`&` **`operator=`**([`TokenizerBuilder`](./TokenizerBuilder.md)`&&` `other`) `noexcept`

</div>

Move-only handle; the moved-from object is empty.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 447

</div>

### \~TokenizerBuilder() {#TokenizerBuilder-3}

<div class="api-signature">

**`~TokenizerBuilder`**()

</div>

Releases the builder (an unbuilt one drops its stages).

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 450

</div>