clika_runtime.tokenizer

Tokenizers: text to token ids and back, chat templates, streaming decode.

:class:Tokenizer loads a tokenizer from a file or a Hugging Face snapshot and encodes, decodes, batches, renders chat templates, and opens a :class:StreamingDecoder for token-by-token text. Its chat methods take the conversation either as JSON text or as a list of {"role": ..., "content": ...} dicts. :class:Encoded is the batch-encoding result (ids, padded or packed tensors, lengths).

Name	Kind
Encoded	class
StreamingDecoder	class
Tokenizer	class
functions	module functions