ClikaRT::tokenizer::PreTokenizer

class

Header: ClikaRT/tokenizer/builder.h

The pre-segmentation stage (optional): cuts text into pieces the model tokenizes independently (whitespace, byte-level, metaspace, regex splits). Composable via sequence(); move-only handle consumed by TokenizerBuilder::pre_tokenizer.

Types

enum PrependScheme

enum class PrependScheme

Where a metaspace pre-tokenizer prepends its marker.

Enumerator	Description
Always
First
Never

Declared in ClikaRT/tokenizer/builder.h, line 198

enum SplitDelimiterBehavior

enum class SplitDelimiterBehavior

How a split keeps or drops the delimiter it matched.

Enumerator	Description
Removed	Drop the matched delimiter.
Isolated	Keep the delimiter as its own piece.
MergedWithPrevious	Glue the delimiter onto the piece before it.
MergedWithNext	Glue the delimiter onto the piece after it.
Contiguous	Merge a RUN of delimiters into one piece.

Declared in ClikaRT/tokenizer/builder.h, line 200

Static member functions

bert()

static PreTokenizer bert()

The BERT pre-tokenizer (whitespace + punctuation).

Declared in ClikaRT/tokenizer/builder.h, line 214

byte_level()

static PreTokenizer byte_level(
    bool add_prefix_space = true,
    bool trim_offsets = true,
    bool use_regex = true
)

GPT-2 byte-level pre-tokenizer. add_prefix_space inserts a leading space; use_regex applies the GPT-2 token regex.

Declared in ClikaRT/tokenizer/builder.h, line 217

digits()

static PreTokenizer digits(bool individual_digits = false)

Split runs of digits (optionally each digit individually).

Declared in ClikaRT/tokenizer/builder.h, line 220

metaspace()

static PreTokenizer metaspace(
    std::string_view replacement = "\xe2\x96\x81",
    PrependScheme prepend_scheme = PrependScheme::Always,
    bool split = true
)

SentencePiece-style metaspace: replace spaces with replacement (default ▁, U+2581). split also breaks on the marker.

Declared in ClikaRT/tokenizer/builder.h, line 223

sequence()

static PreTokenizer sequence(std::vector<PreTokenizer> pretokenizers)

Apply a list of pre-tokenizers in order (each consumed).

Declared in ClikaRT/tokenizer/builder.h, line 227

split()

static PreTokenizer split(
    std::string_view pattern,
    SplitDelimiterBehavior behavior = SplitDelimiterBehavior::Isolated,
    bool invert = false,
    bool is_regex = true
)

Split on pattern. is_regex selects regex vs exact-string; invert splits on the complement; raises if a regex does not compile.

Declared in ClikaRT/tokenizer/builder.h, line 234

whitespace_split()

static PreTokenizer whitespace_split()

Split on whitespace only.

Declared in ClikaRT/tokenizer/builder.h, line 240

Member functions

PreTokenizer(void)

explicit PreTokenizer(void* impl) noexcept

Declared in ClikaRT/tokenizer/builder.h, line 242

PreTokenizer(PreTokenizer)

PreTokenizer(PreTokenizer&& other) noexcept

Move-only handle; the moved-from object is empty.

Declared in ClikaRT/tokenizer/builder.h, line 244

operator=()

PreTokenizer& operator=(PreTokenizer&& other) noexcept

Move-only handle; the moved-from object is empty.

Declared in ClikaRT/tokenizer/builder.h, line 246

~PreTokenizer()

~PreTokenizer()

Releases the pre-tokenizer.

Declared in ClikaRT/tokenizer/builder.h, line 248