Skip to main content

ClikaRT::tokenizer::Token

struct

Header: ClikaRT/tokenizer/tokenizer.h

One token produced by tokenize: its id, the surface string it spells, and the half-open byte span [begin, end) it covers in the source text. special is true for a token the tokenizer injected itself (bos/eos/...), not one that came from the input.

Data members​

id​

std::int32_t id = 0

vocabulary id

Declared in ClikaRT/tokenizer/tokenizer.h, line 139

text​

std::string text

the surface piece (UTF-8)

Declared in ClikaRT/tokenizer/tokenizer.h, line 140

begin​

std::size_t begin = 0

byte span start in the source text

Declared in ClikaRT/tokenizer/tokenizer.h, line 141

end​

std::size_t end = 0

byte span end (half-open)

Declared in ClikaRT/tokenizer/tokenizer.h, line 142

special​

bool special = false

injected by the tokenizer (bos/eos/...), not from the input

Declared in ClikaRT/tokenizer/tokenizer.h, line 143