---
title: "ClikaRT::tokenizer::PreTokenizer"
sidebar_label: "PreTokenizer"
description: "The pre-segmentation stage (optional): cuts text into pieces the model tokenizes independently (whitespace, byte-level, metaspace, regex splits)."
---

<!-- Generated by tools/doxygen2mdx from Doxygen XML. Do not edit. -->

<span class="api-kind api-kind--class">class</span>

**Header:** `ClikaRT/tokenizer/builder.h`

The pre-segmentation stage (optional): cuts text into pieces the model tokenizes independently (whitespace, byte-level, metaspace, regex splits). Composable via [`sequence()`](./PreTokenizer.md#sequence); move-only handle consumed by [`TokenizerBuilder::pre_tokenizer`](./TokenizerBuilder.md#pre_tokenizer).

## Types

### enum PrependScheme {#PrependScheme}

<div class="api-signature">

`enum class` **`PrependScheme`**

</div>

Where a metaspace pre-tokenizer prepends its marker.

| Enumerator | Description |
| --- | --- |
| `Always` |  |
| `First` |  |
| `Never` |  |

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 198

</div>

### enum SplitDelimiterBehavior {#SplitDelimiterBehavior}

<div class="api-signature">

`enum class` **`SplitDelimiterBehavior`**

</div>

How a [`split`](./PreTokenizer.md#split) keeps or drops the delimiter it matched.

| Enumerator | Description |
| --- | --- |
| `Removed` | Drop the matched delimiter. |
| `Isolated` | Keep the delimiter as its own piece. |
| `MergedWithPrevious` | Glue the delimiter onto the piece before it. |
| `MergedWithNext` | Glue the delimiter onto the piece after it. |
| `Contiguous` | Merge a RUN of delimiters into one piece. |

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 200

</div>

## Static member functions

### bert() {#bert}

<div class="api-signature">

`static` [`PreTokenizer`](./PreTokenizer.md) **`bert`**()

</div>

The BERT pre-tokenizer (whitespace + punctuation).

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 214

</div>

### byte\_level() {#byte_level}

<div class="api-signature">

`static` [`PreTokenizer`](./PreTokenizer.md) **`byte_level`**(<br />
&nbsp;&nbsp;&nbsp;&nbsp;`bool` `add_prefix_space` `=` `true`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`bool` `trim_offsets` `=` `true`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`bool` `use_regex` `=` `true`<br />
)

</div>

GPT-2 byte-level pre-tokenizer. `add_prefix_space` inserts a leading space; `use_regex` applies the GPT-2 token regex.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 217

</div>

### digits() {#digits}

<div class="api-signature">

`static` [`PreTokenizer`](./PreTokenizer.md) **`digits`**(`bool` `individual_digits` `=` `false`)

</div>

Split runs of digits (optionally each digit individually).

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 220

</div>

### metaspace() {#metaspace}

<div class="api-signature">

`static` [`PreTokenizer`](./PreTokenizer.md) **`metaspace`**(<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::string_view` `replacement` `=` `"\xe2\x96\x81"`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`PrependScheme`](./PreTokenizer.md#PrependScheme) `prepend_scheme` `=` [`PrependScheme::Always`](./PreTokenizer.md#PrependScheme),<br />
&nbsp;&nbsp;&nbsp;&nbsp;`bool` `split` `=` `true`<br />
)

</div>

SentencePiece-style metaspace: replace spaces with `replacement` (default `▁`, U+2581). [`split`](./PreTokenizer.md#split) also breaks on the marker.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 223

</div>

### sequence() {#sequence}

<div class="api-signature">

`static` [`PreTokenizer`](./PreTokenizer.md) **`sequence`**(`std::vector<`[`PreTokenizer`](./PreTokenizer.md)`>` `pretokenizers`)

</div>

Apply a list of pre-tokenizers in order (each consumed).

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 227

</div>

### split() {#split}

<div class="api-signature">

`static` [`PreTokenizer`](./PreTokenizer.md) **`split`**(<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::string_view` `pattern`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`SplitDelimiterBehavior`](./PreTokenizer.md#SplitDelimiterBehavior) `behavior` `=` [`SplitDelimiterBehavior::Isolated`](./PreTokenizer.md#SplitDelimiterBehavior),<br />
&nbsp;&nbsp;&nbsp;&nbsp;`bool` `invert` `=` `false`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`bool` `is_regex` `=` `true`<br />
)

</div>

Split on `pattern`. `is_regex` selects regex vs exact-string; `invert` splits on the complement; raises if a regex does not compile.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 234

</div>

### whitespace\_split() {#whitespace_split}

<div class="api-signature">

`static` [`PreTokenizer`](./PreTokenizer.md) **`whitespace_split`**()

</div>

Split on whitespace only.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 240

</div>

## Member functions

### PreTokenizer(void) {#PreTokenizer}

<div class="api-signature">

`explicit` **`PreTokenizer`**(`void*` `impl`) `noexcept`

</div>

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 242

</div>

### PreTokenizer(PreTokenizer) {#PreTokenizer-2}

<div class="api-signature">

**`PreTokenizer`**(`PreTokenizer&&` `other`) `noexcept`

</div>

Move-only handle; the moved-from object is empty.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 244

</div>

### operator=() {#operator-assign}

<div class="api-signature">

[`PreTokenizer`](./PreTokenizer.md)`&` **`operator=`**([`PreTokenizer`](./PreTokenizer.md)`&&` `other`) `noexcept`

</div>

Move-only handle; the moved-from object is empty.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 246

</div>

### \~PreTokenizer() {#PreTokenizer-3}

<div class="api-signature">

**`~PreTokenizer`**()

</div>

Releases the pre-tokenizer.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 248

</div>