Skip to main content

tokenizer

family

Declared in clika_rt/clika_rt_core.h.

Members​

NameDescription
clika_rt_tokenizer
tokenizer_apply_chat_template_messages_add_generation_promptClikaRT::tokenizer::Tokenizer::apply_chat_template <- ClikaRT/tokenizer/tokenizer.h one form of the overloaded C++ apply_chat_template; the _messages_add_generation_prompt suffix names this form borrows self messages_json is a JSON document as text (not retained) sized-string protocol (contract §5) C++ defaults (dropped at this surface): add_generation_prompt=true
tokenizer_bos_idClikaRT::tokenizer::Tokenizer::bos_id <- ClikaRT/tokenizer/tokenizer.h borrows self infallible
tokenizer_decodeClikaRT::tokenizer::Tokenizer::decode <- ClikaRT/tokenizer/tokenizer.h borrows self sized-string protocol (contract §5) C++ defaults (dropped at this surface): skip_special_tokens=true
tokenizer_decode_batch_encoded_skip_special_tokensClikaRT::tokenizer::Tokenizer::decode_batch <- ClikaRT/tokenizer/tokenizer.h one form of the overloaded C++ decode_batch; the _encoded_skip_special_tokens suffix names this form borrows self borrows encoded single-owner list: index it with string_list_count / string_list_get_at; string_list_free frees it C++ defaults (dropped at this surface): skip_special_tokens=true
tokenizer_decode_batch_ids_cu_seqlensClikaRT::tokenizer::Tokenizer::decode_batch <- ClikaRT/tokenizer/tokenizer.h one form of the overloaded C++ decode_batch; the _ids_cu_seqlens suffix names this form borrows self borrows ids borrows cu_seqlens single-owner list: index it with string_list_count / string_list_get_at; string_list_free frees it C++ defaults (dropped at this surface): skip_special_tokens=true
tokenizer_decode_batch_ids_skip_special_tokensClikaRT::tokenizer::Tokenizer::decode_batch <- ClikaRT/tokenizer/tokenizer.h one form of the overloaded C++ decode_batch; the _ids_skip_special_tokens suffix names this form borrows self borrows ids single-owner list: index it with string_list_count / string_list_get_at; string_list_free frees it C++ defaults (dropped at this surface): skip_special_tokens=true
tokenizer_encodeClikaRT::tokenizer::Tokenizer::encode <- ClikaRT/tokenizer/tokenizer.h borrows self two-call sizing: buf == NULL sizes; count carries capacity in, required count out (contract §8) C++ defaults (dropped at this surface): add_special_tokens=true
tokenizer_encode_batchClikaRT::tokenizer::Tokenizer::encode_batch <- ClikaRT/tokenizer/tokenizer.h borrows self texts[i] spans texts_lens[i] bytes (the string protocol's sized form, per element) options may be NULL; every option takes its default single-owner handle: the id rows + optional tensor columns ride its getters; encoded_free frees it C++ defaults (dropped at this surface): options={}
tokenizer_encode_chat_messagesClikaRT::tokenizer::Tokenizer::encode_chat <- ClikaRT/tokenizer/tokenizer.h one form of the overloaded C++ encode_chat; the _messages suffix names this form borrows self messages_json is a JSON document as text (not retained) two-call sizing: buf == NULL sizes; count carries capacity in, required count out (contract §8) C++ defaults (dropped at this surface): add_generation_prompt=true, add_special_tokens=false
tokenizer_encode_textClikaRT::tokenizer::Tokenizer::encode_text <- ClikaRT/tokenizer/tokenizer.h borrows self options may be NULL; every option takes its default single-owner handle: the id rows + optional tensor columns ride its getters; encoded_free frees it C++ defaults (dropped at this surface): options={}
tokenizer_eos_idClikaRT::tokenizer::Tokenizer::eos_id <- ClikaRT/tokenizer/tokenizer.h borrows self infallible
tokenizer_free
tokenizer_from_bert_vocabClikaRT::tokenizer::Tokenizer::from_bert_vocab <- ClikaRT/tokenizer/tokenizer.h single-owner handle: tokenizer_free frees it (NULL is a no-op)
tokenizer_from_fileClikaRT::tokenizer::Tokenizer::from_file <- ClikaRT/tokenizer/tokenizer.h single-owner handle: tokenizer_free frees it (NULL is a no-op)
tokenizer_from_gpt2ClikaRT::tokenizer::Tokenizer::from_gpt2 <- ClikaRT/tokenizer/tokenizer.h single-owner handle: tokenizer_free frees it (NULL is a no-op)
tokenizer_from_huggingfaceClikaRT::tokenizer::Tokenizer::from_huggingface <- ClikaRT/tokenizer/tokenizer.h single-owner handle: tokenizer_free frees it (NULL is a no-op)
tokenizer_from_sentencepieceClikaRT::tokenizer::Tokenizer::from_sentencepiece <- ClikaRT/tokenizer/tokenizer.h single-owner handle: tokenizer_free frees it (NULL is a no-op)
tokenizer_from_tekkenClikaRT::tokenizer::Tokenizer::from_tekken <- ClikaRT/tokenizer/tokenizer.h single-owner handle: tokenizer_free frees it (NULL is a no-op)
tokenizer_from_tiktokenClikaRT::tokenizer::Tokenizer::from_tiktoken <- ClikaRT/tokenizer/tokenizer.h single-owner handle: tokenizer_free frees it (NULL is a no-op)
tokenizer_has_chat_templateClikaRT::tokenizer::Tokenizer::has_chat_template <- ClikaRT/tokenizer/tokenizer.h borrows self infallible
tokenizer_id_to_tokenClikaRT::tokenizer::Tokenizer::id_to_token <- ClikaRT/tokenizer/tokenizer.h borrows self sized-string protocol (contract §5)
tokenizer_pad_idClikaRT::tokenizer::Tokenizer::pad_id <- ClikaRT/tokenizer/tokenizer.h borrows self infallible
tokenizer_streaming_decoderClikaRT::tokenizer::Tokenizer::streaming_decoder <- ClikaRT/tokenizer/tokenizer.h borrows self infallible; single-owner handle: streaming_decoder_free frees it (NULL is a no-op) C++ defaults (dropped at this surface): skip_special_tokens=true
tokenizer_token_to_idClikaRT::tokenizer::Tokenizer::token_to_id <- ClikaRT/tokenizer/tokenizer.h borrows self
tokenizer_tokenizeClikaRT::tokenizer::Tokenizer::tokenize <- ClikaRT/tokenizer/tokenizer.h borrows self single-owner list: token_list_count + the per-index field getters; token_list_free frees it C++ defaults (dropped at this surface): add_special_tokens=true
tokenizer_vocab_sizeClikaRT::tokenizer::Tokenizer::vocab_size <- ClikaRT/tokenizer/tokenizer.h borrows self infallible

Type aliases​

using clika_rt_tokenizer​

using clika_rt_tokenizer = struct clika_rt_tokenizer

Declared in clika_rt/clika_rt_core.h, line 327

Functions​

tokenizer_apply_chat_template_messages_add_generation_prompt​

clika_rt_error*(* tokenizer_apply_chat_template_messages_add_generation_prompt)(clika_rt_tokenizer *self, const char *messages_json, size_t messages_json_len, uint8_t add_generation_prompt, char *buf, size_t *len)

ClikaRT::tokenizer::Tokenizer::apply_chat_template <- ClikaRT/tokenizer/tokenizer.h one form of the overloaded C++ apply_chat_template; the _messages_add_generation_prompt suffix names this form borrows self messages_json is a JSON document as text (not retained) sized-string protocol (contract §5) C++ defaults (dropped at this surface): add_generation_prompt=true

Declared in clika_rt/clika_rt_core.h, line 9522

tokenizer_bos_id​

int64_t(* tokenizer_bos_id)(clika_rt_tokenizer *self)

ClikaRT::tokenizer::Tokenizer::bos_id <- ClikaRT/tokenizer/tokenizer.h borrows self infallible

Declared in clika_rt/clika_rt_core.h, line 9479

tokenizer_decode​

clika_rt_error*(* tokenizer_decode)(clika_rt_tokenizer *self, const int32_t *ids, size_t ids_count, uint8_t skip_special_tokens, char *buf, size_t *len)

ClikaRT::tokenizer::Tokenizer::decode <- ClikaRT/tokenizer/tokenizer.h borrows self sized-string protocol (contract §5) C++ defaults (dropped at this surface): skip_special_tokens=true

Declared in clika_rt/clika_rt_core.h, line 9400

tokenizer_decode_batch_encoded_skip_special_tokens​

clika_rt_error*(* tokenizer_decode_batch_encoded_skip_special_tokens)(clika_rt_tokenizer *self, const clika_rt_encoded *encoded, uint8_t skip_special_tokens, clika_rt_string_list **out)

ClikaRT::tokenizer::Tokenizer::decode_batch <- ClikaRT/tokenizer/tokenizer.h one form of the overloaded C++ decode_batch; the _encoded_skip_special_tokens suffix names this form borrows self borrows encoded single-owner list: index it with string_list_count / string_list_get_at; string_list_free frees it C++ defaults (dropped at this surface): skip_special_tokens=true

Declared in clika_rt/clika_rt_core.h, line 9447

tokenizer_decode_batch_ids_cu_seqlens​

clika_rt_error*(* tokenizer_decode_batch_ids_cu_seqlens)(clika_rt_tokenizer *self, clika_rt_tensor *ids, clika_rt_tensor *cu_seqlens, uint8_t skip_special_tokens, clika_rt_string_list **out)

ClikaRT::tokenizer::Tokenizer::decode_batch <- ClikaRT/tokenizer/tokenizer.h one form of the overloaded C++ decode_batch; the _ids_cu_seqlens suffix names this form borrows self borrows ids borrows cu_seqlens single-owner list: index it with string_list_count / string_list_get_at; string_list_free frees it C++ defaults (dropped at this surface): skip_special_tokens=true

Declared in clika_rt/clika_rt_core.h, line 9470

tokenizer_decode_batch_ids_skip_special_tokens​

clika_rt_error*(* tokenizer_decode_batch_ids_skip_special_tokens)(clika_rt_tokenizer *self, clika_rt_tensor *ids, uint8_t skip_special_tokens, clika_rt_string_list **out)

ClikaRT::tokenizer::Tokenizer::decode_batch <- ClikaRT/tokenizer/tokenizer.h one form of the overloaded C++ decode_batch; the _ids_skip_special_tokens suffix names this form borrows self borrows ids single-owner list: index it with string_list_count / string_list_get_at; string_list_free frees it C++ defaults (dropped at this surface): skip_special_tokens=true

Declared in clika_rt/clika_rt_core.h, line 9458

tokenizer_encode​

clika_rt_error*(* tokenizer_encode)(clika_rt_tokenizer *self, const char *text, size_t text_len, uint8_t add_special_tokens, int32_t *buf, size_t *count)

ClikaRT::tokenizer::Tokenizer::encode <- ClikaRT/tokenizer/tokenizer.h borrows self two-call sizing: buf == NULL sizes; count carries capacity in, required count out (contract §8) C++ defaults (dropped at this surface): add_special_tokens=true

Declared in clika_rt/clika_rt_core.h, line 9389

tokenizer_encode_batch​

clika_rt_error*(* tokenizer_encode_batch)(clika_rt_tokenizer *self, const char *const *texts, const size_t *texts_lens, size_t texts_count, const clika_rt_encode_options *options, clika_rt_encoded **out)

ClikaRT::tokenizer::Tokenizer::encode_batch <- ClikaRT/tokenizer/tokenizer.h borrows self texts[i] spans texts_lens[i] bytes (the string protocol's sized form, per element) options may be NULL; every option takes its default single-owner handle: the id rows + optional tensor columns ride its getters; encoded_free frees it C++ defaults (dropped at this surface): options={}

Declared in clika_rt/clika_rt_core.h, line 9434

tokenizer_encode_chat_messages​

clika_rt_error*(* tokenizer_encode_chat_messages)(clika_rt_tokenizer *self, const char *messages_json, size_t messages_json_len, uint8_t add_generation_prompt, uint8_t add_special_tokens, int32_t *buf, size_t *count)

ClikaRT::tokenizer::Tokenizer::encode_chat <- ClikaRT/tokenizer/tokenizer.h one form of the overloaded C++ encode_chat; the _messages suffix names this form borrows self messages_json is a JSON document as text (not retained) two-call sizing: buf == NULL sizes; count carries capacity in, required count out (contract §8) C++ defaults (dropped at this surface): add_generation_prompt=true, add_special_tokens=false

Declared in clika_rt/clika_rt_core.h, line 9535

tokenizer_encode_text​

clika_rt_error*(* tokenizer_encode_text)(clika_rt_tokenizer *self, const char *text, size_t text_len, const clika_rt_encode_options *options, clika_rt_encoded **out)

ClikaRT::tokenizer::Tokenizer::encode_text <- ClikaRT/tokenizer/tokenizer.h borrows self options may be NULL; every option takes its default single-owner handle: the id rows + optional tensor columns ride its getters; encoded_free frees it C++ defaults (dropped at this surface): options={}

Declared in clika_rt/clika_rt_core.h, line 9422

tokenizer_eos_id​

int64_t(* tokenizer_eos_id)(clika_rt_tokenizer *self)

ClikaRT::tokenizer::Tokenizer::eos_id <- ClikaRT/tokenizer/tokenizer.h borrows self infallible

Declared in clika_rt/clika_rt_core.h, line 9484

tokenizer_free​

void(* tokenizer_free)(clika_rt_tokenizer *h)

Declared in clika_rt/clika_rt_core.h, line 723

tokenizer_from_bert_vocab​

clika_rt_error*(* tokenizer_from_bert_vocab)(const char *dir, size_t dir_len, clika_rt_tokenizer **out)

ClikaRT::tokenizer::Tokenizer::from_bert_vocab <- ClikaRT/tokenizer/tokenizer.h single-owner handle: tokenizer_free frees it (NULL is a no-op)

Declared in clika_rt/clika_rt_core.h, line 9375

tokenizer_from_file​

clika_rt_error*(* tokenizer_from_file)(const char *path, size_t path_len, clika_rt_tokenizer **out)

ClikaRT::tokenizer::Tokenizer::from_file <- ClikaRT/tokenizer/tokenizer.h single-owner handle: tokenizer_free frees it (NULL is a no-op)

Declared in clika_rt/clika_rt_core.h, line 9381

tokenizer_from_gpt2​

clika_rt_error*(* tokenizer_from_gpt2)(const char *dir, size_t dir_len, clika_rt_tokenizer **out)

ClikaRT::tokenizer::Tokenizer::from_gpt2 <- ClikaRT/tokenizer/tokenizer.h single-owner handle: tokenizer_free frees it (NULL is a no-op)

Declared in clika_rt/clika_rt_core.h, line 9363

tokenizer_from_huggingface​

clika_rt_error*(* tokenizer_from_huggingface)(const char *path, size_t path_len, clika_rt_tokenizer **out)

ClikaRT::tokenizer::Tokenizer::from_huggingface <- ClikaRT/tokenizer/tokenizer.h single-owner handle: tokenizer_free frees it (NULL is a no-op)

Declared in clika_rt/clika_rt_core.h, line 9345

tokenizer_from_sentencepiece​

clika_rt_error*(* tokenizer_from_sentencepiece)(const char *path, size_t path_len, clika_rt_tokenizer **out)

ClikaRT::tokenizer::Tokenizer::from_sentencepiece <- ClikaRT/tokenizer/tokenizer.h single-owner handle: tokenizer_free frees it (NULL is a no-op)

Declared in clika_rt/clika_rt_core.h, line 9351

tokenizer_from_tekken​

clika_rt_error*(* tokenizer_from_tekken)(const char *path, size_t path_len, clika_rt_tokenizer **out)

ClikaRT::tokenizer::Tokenizer::from_tekken <- ClikaRT/tokenizer/tokenizer.h single-owner handle: tokenizer_free frees it (NULL is a no-op)

Declared in clika_rt/clika_rt_core.h, line 9357

tokenizer_from_tiktoken​

clika_rt_error*(* tokenizer_from_tiktoken)(const char *path, size_t path_len, clika_rt_tokenizer **out)

ClikaRT::tokenizer::Tokenizer::from_tiktoken <- ClikaRT/tokenizer/tokenizer.h single-owner handle: tokenizer_free frees it (NULL is a no-op)

Declared in clika_rt/clika_rt_core.h, line 9369

tokenizer_has_chat_template​

uint8_t(* tokenizer_has_chat_template)(clika_rt_tokenizer *self)

ClikaRT::tokenizer::Tokenizer::has_chat_template <- ClikaRT/tokenizer/tokenizer.h borrows self infallible

Declared in clika_rt/clika_rt_core.h, line 9514

tokenizer_id_to_token​

clika_rt_error*(* tokenizer_id_to_token)(clika_rt_tokenizer *self, int32_t id, char *buf, size_t *len)

ClikaRT::tokenizer::Tokenizer::id_to_token <- ClikaRT/tokenizer/tokenizer.h borrows self sized-string protocol (contract §5)

Declared in clika_rt/clika_rt_core.h, line 9506

tokenizer_pad_id​

int64_t(* tokenizer_pad_id)(clika_rt_tokenizer *self)

ClikaRT::tokenizer::Tokenizer::pad_id <- ClikaRT/tokenizer/tokenizer.h borrows self infallible

Declared in clika_rt/clika_rt_core.h, line 9489

tokenizer_streaming_decoder​

clika_rt_streaming_decoder*(* tokenizer_streaming_decoder)(clika_rt_tokenizer *self, uint8_t skip_special_tokens)

ClikaRT::tokenizer::Tokenizer::streaming_decoder <- ClikaRT/tokenizer/tokenizer.h borrows self infallible; single-owner handle: streaming_decoder_free frees it (NULL is a no-op) C++ defaults (dropped at this surface): skip_special_tokens=true

Declared in clika_rt/clika_rt_core.h, line 9547

tokenizer_token_to_id​

clika_rt_error*(* tokenizer_token_to_id)(clika_rt_tokenizer *self, const char *token, size_t token_len, int32_t *out)

ClikaRT::tokenizer::Tokenizer::token_to_id <- ClikaRT/tokenizer/tokenizer.h borrows self

Declared in clika_rt/clika_rt_core.h, line 9498

tokenizer_tokenize​

clika_rt_error*(* tokenizer_tokenize)(clika_rt_tokenizer *self, const char *text, size_t text_len, uint8_t add_special_tokens, clika_rt_token_list **out)

ClikaRT::tokenizer::Tokenizer::tokenize <- ClikaRT/tokenizer/tokenizer.h borrows self single-owner list: token_list_count + the per-index field getters; token_list_free frees it C++ defaults (dropped at this surface): add_special_tokens=true

Declared in clika_rt/clika_rt_core.h, line 9411

tokenizer_vocab_size​

int64_t(* tokenizer_vocab_size)(clika_rt_tokenizer *self)

ClikaRT::tokenizer::Tokenizer::vocab_size <- ClikaRT/tokenizer/tokenizer.h borrows self infallible

Declared in clika_rt/clika_rt_core.h, line 9494