Skip to main content

ClikaRT::tokenizer::Decoder

class

Header: ClikaRT/tokenizer/builder.h

The detokenization stage (optional): turns token strings back into text (reverse byte-level / metaspace / WordPiece, fuse, strip). Composable via sequence(); move-only handle consumed by TokenizerBuilder::decoder.

Static member functions​

byte_level()​

static Decoder byte_level()

Reverse the GPT-2 byte-level map.

Declared in ClikaRT/tokenizer/builder.h, line 336

metaspace()​

static Decoder metaspace(char32_t replacement = U '▁', bool prepend = true)

Reverse metaspace: turn the replacement marker codepoint back into a space (default ▁, U+2581); prepend drops the leading marker.

Declared in ClikaRT/tokenizer/builder.h, line 339

wordpiece()​

static Decoder wordpiece(std::string_view prefix = "##", bool cleanup = true)

Reverse WordPiece: strip prefix (##) and join; cleanup fixes spacing around punctuation.

Declared in ClikaRT/tokenizer/builder.h, line 342

replace()​

static Decoder replace(
    std::string_view pattern,
    std::string_view content,
    bool is_regex = false
)

Replace pattern with content on each token. is_regex selects regex vs exact-string; raises if a regex does not compile.

Declared in ClikaRT/tokenizer/builder.h, line 347

strip()​

static Decoder strip(
    char32_t content,
    std::int64_t start,
    std::int64_t stop
)

Trim up to start leading and stop trailing occurrences of the content codepoint from each token.

Declared in ClikaRT/tokenizer/builder.h, line 353

fuse()​

static Decoder fuse()

Concatenate every token into one (no separator).

Declared in ClikaRT/tokenizer/builder.h, line 355

byte_fallback()​

static Decoder byte_fallback()

Decode an unknown-byte token (<0x0A>) back into its raw byte.

Declared in ClikaRT/tokenizer/builder.h, line 357

sequence()​

static Decoder sequence(std::vector<Decoder> decoders)

Apply a list of decoders in order (each consumed).

Declared in ClikaRT/tokenizer/builder.h, line 359

Member functions​

Decoder(void)​

explicit Decoder(void* impl) noexcept

Declared in ClikaRT/tokenizer/builder.h, line 361

Decoder(Decoder)​

Decoder(Decoder&& other) noexcept

Move-only handle; the moved-from object is empty.

Declared in ClikaRT/tokenizer/builder.h, line 363

operator=()​

Decoder& operator=(Decoder&& other) noexcept

Move-only handle; the moved-from object is empty.

Declared in ClikaRT/tokenizer/builder.h, line 365

~Decoder()​

~Decoder()

Releases the decoder.

Declared in ClikaRT/tokenizer/builder.h, line 367