Skip to main content

ClikaRT::tokenizer::BpeOptions

struct

Header: ClikaRT/tokenizer/builder.h

Extra BPE knobs; every field is optional and defaults to HF's behavior.

Data members​

unk_token​

std::string_view unk_token

Unknown-token string; unset means unknown pieces are dropped.

Declared in ClikaRT/tokenizer/builder.h, line 50

has_unk_token​

bool has_unk_token = false

Presence flag for unk_token.

Declared in ClikaRT/tokenizer/builder.h, line 52

continuing_subword_prefix​

std::string_view continuing_subword_prefix

Prefix marking a continuing subword (WordPiece-style ##); unset = none.

Declared in ClikaRT/tokenizer/builder.h, line 54

has_continuing_subword_prefix​

bool has_continuing_subword_prefix = false

Presence flag for continuing_subword_prefix.

Declared in ClikaRT/tokenizer/builder.h, line 56

end_of_word_suffix​

std::string_view end_of_word_suffix

Suffix marking the end of a word; unset = none.

Declared in ClikaRT/tokenizer/builder.h, line 58

has_end_of_word_suffix​

bool has_end_of_word_suffix = false

Presence flag for end_of_word_suffix.

Declared in ClikaRT/tokenizer/builder.h, line 60

fuse_unk​

bool fuse_unk = false

Fuse consecutive unknown pieces into ONE unk token.

Declared in ClikaRT/tokenizer/builder.h, line 63

byte_fallback​

bool byte_fallback = false

Decompose an unknown piece into byte tokens instead of unk.

Declared in ClikaRT/tokenizer/builder.h, line 66

ignore_merges​

bool ignore_merges = false

Skip the merge loop for words already in the vocabulary.

Declared in ClikaRT/tokenizer/builder.h, line 69