clika_runtime.tokenizer

Tokenizers: text to token ids and back, chat templates, streaming decode, and a builder for a tokenizer with no file.

:class:Tokenizer loads a tokenizer from a file or a Hugging Face snapshot and encodes, decodes, batches, renders chat templates, and opens a :class:StreamingDecoder for token-by-token text. Its chat methods take the conversation either as JSON text or as a list of {"role": ..., "content": ...} dicts. :class:Encoded is the batch-encoding result (ids, padded or packed tensors, lengths). :class:TokenizerBuilder assembles a tokenizer from an explicit vocabulary (a :class:Model: BPE, unigram, word level or WordPiece) and the optional :class:Normalizer, :class:PreTokenizer, :class:PostProcessor and :class:Decoder stages.

Name	Kind
Decoder	class
Encoded	class
Model	class
Normalizer	class
PostProcessor	class
PreTokenizer	class
StreamingDecoder	class
Tokenizer	class
TokenizerBuilder	class
functions	module functions