Skip to main content

ClikaRT::tokenizer::PostProcessor::SpecialToken

struct

Header: ClikaRT/tokenizer/builder.h

A special token referenced by a template: its key maps to a parallel (tokens, ids) pair.

Data members​

key​

std::string_view key

The key template pieces reference.

Declared in ClikaRT/tokenizer/builder.h, line 279

tokens​

std::vector<std::string_view> tokens

The token strings emitted for this key. The vector OWNS the list (assigning from a braced list is safe); the referenced STRINGS must stay alive until the build call, like every string_view input on this builder.

Declared in ClikaRT/tokenizer/builder.h, line 284

ids​

std::vector<std::int32_t> ids

The ids emitted alongside tokens (parallel, same length). The vector owns its values.

Declared in ClikaRT/tokenizer/builder.h, line 287