Skip to main content

token_list

family

Declared in clika_rt/clika_rt_core.h.

Members​

NameDescription
clika_rt_token_list
token_list_begin_atbyte span start in the source text
token_list_count
token_list_end_atbyte span end (half-open)
token_list_free
token_list_id_atvocabulary id (0 when i is out of range)
token_list_special_at1 when the token was injected by the tokenizer (bos/eos/...)
token_list_text_atthe surface piece (UTF-8), sized-string protocol

Type aliases​

using clika_rt_token_list​

using clika_rt_token_list = struct clika_rt_token_list

Declared in clika_rt/clika_rt_core.h, line 326

Functions​

token_list_begin_at​

size_t(* token_list_begin_at)(const clika_rt_token_list *l, size_t i)

byte span start in the source text

Declared in clika_rt/clika_rt_core.h, line 723

token_list_count​

size_t(* token_list_count)(const clika_rt_token_list *l)

Declared in clika_rt/clika_rt_core.h, line 713

token_list_end_at​

size_t(* token_list_end_at)(const clika_rt_token_list *l, size_t i)

byte span end (half-open)

Declared in clika_rt/clika_rt_core.h, line 726

token_list_free​

void(* token_list_free)(clika_rt_token_list *l)

Declared in clika_rt/clika_rt_core.h, line 731

token_list_id_at​

int32_t(* token_list_id_at)(const clika_rt_token_list *l, size_t i)

vocabulary id (0 when i is out of range)

Declared in clika_rt/clika_rt_core.h, line 715

token_list_special_at​

uint8_t(* token_list_special_at)(const clika_rt_token_list *l, size_t i)

1 when the token was injected by the tokenizer (bos/eos/...)

Declared in clika_rt/clika_rt_core.h, line 729

token_list_text_at​

clika_rt_status(* token_list_text_at)(const clika_rt_token_list *l, size_t i, char *buf, size_t *len)

the surface piece (UTF-8), sized-string protocol

Declared in clika_rt/clika_rt_core.h, line 718