Skip to main content

ClikaRT::tokenizer::Model

class

Header: ClikaRT/tokenizer/builder.h

The core tokenization stage (required): BPE / Unigram / WordLevel / WordPiece, built from an explicit vocabulary; no file needed. Consumed by TokenizerBuilder's constructor; move-only handle.

Static member functions​

bpe()​

static Model bpe(
    Span<const VocabEntry> vocab,
    Span<const MergeRule> merges,
    const BpeOptions& options = {}
)

Byte-pair encoding from a vocabulary and an ordered merge list (rank = position). Raises if a merge references a token missing from the vocab.

Declared in ClikaRT/tokenizer/builder.h, line 85

unigram()​

static Model unigram(
    Span<const UnigramEntry> vocab,
    std::int64_t unk_id = -1,
    bool has_unk_id = false,
    bool byte_fallback = false
)

Unigram language model. unk_id (if set) names the unknown token's id; byte_fallback decomposes an unknown piece into byte tokens.

Declared in ClikaRT/tokenizer/builder.h, line 94

word_level()​

static Model word_level(Span<const VocabEntry> vocab, std::string_view unk_token)

Whole-word vocabulary lookup; an out-of-vocabulary word maps to unk_token.

Declared in ClikaRT/tokenizer/builder.h, line 102

word_piece()​

static Model word_piece(
    Span<const VocabEntry> vocab,
    std::string_view unk_token,
    std::string_view continuing_subword_prefix = "##",
    std::int64_t max_input_chars_per_word = 100
)

Greedy longest-match WordPiece. continuing_subword_prefix (e.g. ##) marks a non-initial subword; a word longer than max_input_chars_per_word maps to unk_token.

Declared in ClikaRT/tokenizer/builder.h, line 113

Member functions​

Model(void)​

explicit Model(void* impl) noexcept

Declared in ClikaRT/tokenizer/builder.h, line 121

Model(Model)​

Model(Model&& other) noexcept

Move-only handle; the moved-from object is empty.

Declared in ClikaRT/tokenizer/builder.h, line 123

operator=()​

Model& operator=(Model&& other) noexcept

Move-only handle; the moved-from object is empty.

Declared in ClikaRT/tokenizer/builder.h, line 125

~Model()​

~Model()

Releases the model.

Declared in ClikaRT/tokenizer/builder.h, line 127