---
title: "ClikaRT::tokenizer::BpeOptions"
sidebar_label: "BpeOptions"
description: "Extra BPE knobs; every field is optional and defaults to HF's behavior."
---

<!-- Generated by tools/doxygen2mdx from Doxygen XML. Do not edit. -->

<span class="api-kind api-kind--struct">struct</span>

**Header:** `ClikaRT/tokenizer/builder.h`

Extra BPE knobs; every field is optional and defaults to HF's behavior.

## Data members

### unk\_token {#unk_token}

<div class="api-signature">

`std::string_view` **`unk_token`**

</div>

Unknown-token string; unset means unknown pieces are dropped.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 50

</div>

### has\_unk\_token {#has_unk_token}

<div class="api-signature">

`bool` **`has_unk_token`** `= false`

</div>

Presence flag for [`unk_token`](./BpeOptions.md#unk_token).

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 52

</div>

### continuing\_subword\_prefix {#continuing_subword_prefix}

<div class="api-signature">

`std::string_view` **`continuing_subword_prefix`**

</div>

Prefix marking a continuing subword (WordPiece-style `##`); unset = none.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 54

</div>

### has\_continuing\_subword\_prefix {#has_continuing_subword_prefix}

<div class="api-signature">

`bool` **`has_continuing_subword_prefix`** `= false`

</div>

Presence flag for [`continuing_subword_prefix`](./BpeOptions.md#continuing_subword_prefix).

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 56

</div>

### end\_of\_word\_suffix {#end_of_word_suffix}

<div class="api-signature">

`std::string_view` **`end_of_word_suffix`**

</div>

Suffix marking the end of a word; unset = none.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 58

</div>

### has\_end\_of\_word\_suffix {#has_end_of_word_suffix}

<div class="api-signature">

`bool` **`has_end_of_word_suffix`** `= false`

</div>

Presence flag for [`end_of_word_suffix`](./BpeOptions.md#end_of_word_suffix).

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 60

</div>

### fuse\_unk {#fuse_unk}

<div class="api-signature">

`bool` **`fuse_unk`** `= false`

</div>

Fuse consecutive unknown pieces into ONE unk token.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 63

</div>

### byte\_fallback {#byte_fallback}

<div class="api-signature">

`bool` **`byte_fallback`** `= false`

</div>

Decompose an unknown piece into byte tokens instead of unk.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 66

</div>

### ignore\_merges {#ignore_merges}

<div class="api-signature">

`bool` **`ignore_merges`** `= false`

</div>

Skip the merge loop for words already in the vocabulary.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 69

</div>