Skip to main content

Tokenizer

bos_id (property)​

The begin-of-sequence id; -1 when the config names none.

eos_id (property)​

The end-of-sequence id; -1 when the config names none.

has_chat_template (property)​

(self) -> bool

pad_id (property)​

The padding id; -1 when the config names none.

vocab_size (property)​

(self) -> int

__init__​

__init__(self, /, *args, **kwargs)

Initialize self. See help(type(self)) for accurate signature.

apply_chat_template​

apply_chat_template(self: 'Tokenizer', messages: 'Messages', add_generation_prompt: 'bool' = True) -> 'str'

Render the chat template over the conversation.

messages is a list of {role, content} dicts, or the same conversation as JSON text. Returns the prompt string.

decode​

decodedecode(self, ids: collections.abc.Sequence[int], skip_special_tokens: bool = True) -> str

decode(self, ids: collections.abc.Sequence[int], skip_special_tokens: bool = True) -> str

Decode token ids back into text.

decode_batch​

decode_batchdecode_batch(self, sequences: collections.abc.Sequence[collections.abc.Sequence[int]], skip_special_tokens: bool = True) -> list[str]

decode_batch(self, sequences: collections.abc.Sequence[collections.abc.Sequence[int]], skip_special_tokens: bool = True) -> list[str]

Decode many id sequences, one string per sequence, in order.

encode​

encodeencode(self, text: str, add_special_tokens: bool = True) -> list[int]

encode(self, text: str, add_special_tokens: bool = True) -> list[int]

Encode text into token ids.

encode_batch​

encode_batchencode_batch(self, texts: collections.abc.Sequence[str]) -> clika_runtime._core.tokenizer.Encoded

encode_batch(self, texts: collections.abc.Sequence[str]) -> clika_runtime._core.tokenizer.Encoded

Encode a batch of texts into tensor form: a padded [B, S] input_ids + attention_mask Encoded record.

encode_chat​

encode_chat(self: 'Tokenizer', messages: 'Messages', add_generation_prompt: 'bool' = True, add_special_tokens: 'bool' = True) -> 'list[int]'

Render the chat template and encode the result in one call.

messages takes the same two forms as apply_chat_template.

from_file​

from_file(*args, **kwargs)

from_file(path: str) -> clika_runtime._core.tokenizer.Tokenizer

Load any supported tokenizer artifact (file or directory), detecting the format: tokenizer.json, SentencePiece, tekken, vocab.json+merges.txt, tiktoken. A sibling tokenizer_config.json (bos/eos + chat template) overlays when present.

from_huggingface​

from_huggingface(*args, **kwargs)

from_huggingface(path: str) -> clika_runtime._core.tokenizer.Tokenizer

Load a tokenizer.json (or a directory holding one), with the tokenizer_config.json overlay.

id_to_token​

id_to_tokenid_to_token(self, token_id: int) -> str

id_to_token(self, token_id: int) -> str

The surface token of a vocabulary id.

streaming_decoder​

streaming_decoderstreaming_decoder(self, skip_special_tokens: bool = True) -> clika_runtime._core.tokenizer.StreamingDecoder

streaming_decoder(self, skip_special_tokens: bool = True) -> clika_runtime._core.tokenizer.StreamingDecoder

A per-id incremental decoder for generation loops; the handle stays valid even after this Tokenizer is destroyed.

token_to_id​

token_to_idtoken_to_id(self, token: str) -> int

token_to_id(self, token: str) -> int

The vocabulary id of a surface token.