---
title: "ClikaRT::tokenizer::Model"
sidebar_label: "Model"
description: "The core tokenization stage (required): BPE / Unigram / WordLevel / WordPiece, built from an explicit vocabulary; no file needed. Consumed by TokenizerBuilder's constructor; move-only handle."
---

<!-- Generated by tools/doxygen2mdx from Doxygen XML. Do not edit. -->

<span class="api-kind api-kind--class">class</span>

**Header:** `ClikaRT/tokenizer/builder.h`

The core tokenization stage (required): BPE / Unigram / WordLevel / WordPiece, built from an explicit vocabulary; no file needed. Consumed by [`TokenizerBuilder`](./TokenizerBuilder.md)'s constructor; move-only handle.

## Static member functions

### bpe() {#bpe}

<div class="api-signature">

`static` [`Model`](./Model.md) **`bpe`**(<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`Span`](../Span.md)`<const` [`VocabEntry`](./index.md#VocabEntry)`>` `vocab`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`Span`](../Span.md)`<const` [`MergeRule`](./index.md#MergeRule)`>` `merges`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`const` [`BpeOptions`](./BpeOptions.md)`&` `options` `=` `{}`<br />
)

</div>

Byte-pair encoding from a vocabulary and an ordered merge list (rank = position). Raises if a merge references a token missing from the vocab.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 85

</div>

### unigram() {#unigram}

<div class="api-signature">

`static` [`Model`](./Model.md) **`unigram`**(<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`Span`](../Span.md)`<const` [`UnigramEntry`](./index.md#UnigramEntry)`>` `vocab`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::int64_t` `unk_id` `=` `-1`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`bool` `has_unk_id` `=` `false`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`bool` `byte_fallback` `=` `false`<br />
)

</div>

Unigram language model. `unk_id` (if set) names the unknown token's id; `byte_fallback` decomposes an unknown piece into byte tokens.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 94

</div>

### word\_level() {#word_level}

<div class="api-signature">

`static` [`Model`](./Model.md) **`word_level`**([`Span`](../Span.md)`<const` [`VocabEntry`](./index.md#VocabEntry)`>` `vocab`, `std::string_view` `unk_token`)

</div>

Whole-word vocabulary lookup; an out-of-vocabulary word maps to `unk_token`.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 102

</div>

### word\_piece() {#word_piece}

<div class="api-signature">

`static` [`Model`](./Model.md) **`word_piece`**(<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`Span`](../Span.md)`<const` [`VocabEntry`](./index.md#VocabEntry)`>` `vocab`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::string_view` `unk_token`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::string_view` `continuing_subword_prefix` `=` `"##"`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::int64_t` `max_input_chars_per_word` `=` `100`<br />
)

</div>

Greedy longest-match WordPiece. `continuing_subword_prefix` (e.g. `##`) marks a non-initial subword; a word longer than `max_input_chars_per_word` maps to `unk_token`.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 113

</div>

## Member functions

### Model(void) {#Model}

<div class="api-signature">

`explicit` **`Model`**(`void*` `impl`) `noexcept`

</div>

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 121

</div>

### Model(Model) {#Model-2}

<div class="api-signature">

**`Model`**(`Model&&` `other`) `noexcept`

</div>

Move-only handle; the moved-from object is empty.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 123

</div>

### operator=() {#operator-assign}

<div class="api-signature">

[`Model`](./Model.md)`&` **`operator=`**([`Model`](./Model.md)`&&` `other`) `noexcept`

</div>

Move-only handle; the moved-from object is empty.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 125

</div>

### \~Model() {#Model-3}

<div class="api-signature">

**`~Model`**()

</div>

Releases the model.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 127

</div>