---
title: "ClikaRT::tokenizer::Decoder"
sidebar_label: "Decoder"
description: "The detokenization stage (optional): turns token strings back into text (reverse byte-level / metaspace / WordPiece, fuse, strip)."
---

<!-- Generated by tools/doxygen2mdx from Doxygen XML. Do not edit. -->

<span class="api-kind api-kind--class">class</span>

**Header:** `ClikaRT/tokenizer/builder.h`

The detokenization stage (optional): turns token strings back into text (reverse byte-level / metaspace / WordPiece, fuse, strip). Composable via [`sequence()`](./Decoder.md#sequence); move-only handle consumed by [`TokenizerBuilder::decoder`](./TokenizerBuilder.md#decoder).

## Static member functions

### byte\_level() {#byte_level}

<div class="api-signature">

`static` [`Decoder`](./Decoder.md) **`byte_level`**()

</div>

Reverse the GPT-2 byte-level map.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 336

</div>

### metaspace() {#metaspace}

<div class="api-signature">

`static` [`Decoder`](./Decoder.md) **`metaspace`**(`char32_t` `replacement` `=` `U '▁'`, `bool` `prepend` `=` `true`)

</div>

Reverse metaspace: turn the `replacement` marker codepoint back into a space (default `▁`, U+2581); `prepend` drops the leading marker.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 339

</div>

### wordpiece() {#wordpiece}

<div class="api-signature">

`static` [`Decoder`](./Decoder.md) **`wordpiece`**(`std::string_view` `prefix` `=` `"##"`, `bool` `cleanup` `=` `true`)

</div>

Reverse WordPiece: strip `prefix` (`##`) and join; `cleanup` fixes spacing around punctuation.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 342

</div>

### replace() {#replace}

<div class="api-signature">

`static` [`Decoder`](./Decoder.md) **`replace`**(<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::string_view` `pattern`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::string_view` `content`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`bool` `is_regex` `=` `false`<br />
)

</div>

Replace `pattern` with `content` on each token. `is_regex` selects regex vs exact-string; raises if a regex does not compile.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 347

</div>

### strip() {#strip}

<div class="api-signature">

`static` [`Decoder`](./Decoder.md) **`strip`**(<br />
&nbsp;&nbsp;&nbsp;&nbsp;`char32_t` `content`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::int64_t` `start`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::int64_t` `stop`<br />
)

</div>

Trim up to `start` leading and `stop` trailing occurrences of the `content` codepoint from each token.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 353

</div>

### fuse() {#fuse}

<div class="api-signature">

`static` [`Decoder`](./Decoder.md) **`fuse`**()

</div>

Concatenate every token into one (no separator).

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 355

</div>

### byte\_fallback() {#byte_fallback}

<div class="api-signature">

`static` [`Decoder`](./Decoder.md) **`byte_fallback`**()

</div>

Decode an unknown-byte token (`<0x0A>`) back into its raw byte.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 357

</div>

### sequence() {#sequence}

<div class="api-signature">

`static` [`Decoder`](./Decoder.md) **`sequence`**(`std::vector<`[`Decoder`](./Decoder.md)`>` `decoders`)

</div>

Apply a list of decoders in order (each consumed).

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 359

</div>

## Member functions

### Decoder(void) {#Decoder}

<div class="api-signature">

`explicit` **`Decoder`**(`void*` `impl`) `noexcept`

</div>

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 361

</div>

### Decoder(Decoder) {#Decoder-2}

<div class="api-signature">

**`Decoder`**(`Decoder&&` `other`) `noexcept`

</div>

Move-only handle; the moved-from object is empty.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 363

</div>

### operator=() {#operator-assign}

<div class="api-signature">

[`Decoder`](./Decoder.md)`&` **`operator=`**([`Decoder`](./Decoder.md)`&&` `other`) `noexcept`

</div>

Move-only handle; the moved-from object is empty.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 365

</div>

### \~Decoder() {#Decoder-3}

<div class="api-signature">

**`~Decoder`**()

</div>

Releases the decoder.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 367

</div>