Api: token_list

Api members in the token_list family.

Member
`token_list_begin_at`
`token_list_count`
`token_list_end_at`
`token_list_free`
`token_list_id_at`
`token_list_special_at`
`token_list_text_at` / `f_token_list_text_at`

`token_list_begin_at`

fn token_list_begin_at(self, l: &TokenList, i: usize) -> usize

Wraps api-table member token_list_begin_at. byte span start in the source text

fn token_list_count(self, l: &TokenList) -> usize

Wraps api-table member token_list_count.

fn token_list_end_at(self, l: &TokenList, i: usize) -> usize

Wraps api-table member token_list_end_at. byte span end (half-open)

fn token_list_free(self, l: &TokenList)

Wraps api-table member token_list_free.

fn token_list_id_at(self, l: &TokenList, i: usize) -> i32

Wraps api-table member token_list_id_at. vocabulary id (0 when i is out of range)

fn token_list_special_at(self, l: &TokenList, i: usize) -> bool

Wraps api-table member token_list_special_at. 1 when the token was injected by the tokenizer (bos/eos/...)

fn f_token_list_text_at(self, l: &TokenList, i: usize) -> Result<String>
fn token_list_text_at(self, l: &TokenList, i: usize) -> String

Wraps api-table member token_list_text_at. the surface piece (UTF-8), sized-string protocol