---
title: "clika_runtime.tokenizer"
sidebar_label: "clika_runtime.tokenizer"
description: "The clika_runtime.tokenizer module, introspected from the installed wheel."
---

<!-- Generated by tools/api_reference/generate_api_docs.py. Do not edit. -->

Tokenizers: text to token ids and back, chat templates, streaming decode,
and a builder for a tokenizer with no file.

:class:`Tokenizer` loads a tokenizer from a file or a Hugging Face
snapshot and encodes, decodes, batches, renders chat templates, and opens
a :class:`StreamingDecoder` for token-by-token text. Its chat methods take
the conversation either as JSON text or as a list of ``{"role": ...,
"content": ...}`` dicts. :class:`Encoded` is the batch-encoding result
(ids, padded or packed tensors, lengths). :class:`TokenizerBuilder`
assembles a tokenizer from an explicit vocabulary (a :class:`Model`: BPE,
unigram, word level or WordPiece) and the optional :class:`Normalizer`,
:class:`PreTokenizer`, :class:`PostProcessor` and :class:`Decoder` stages.

| Name | Kind |
| --- | --- |
| [`Decoder`](./Decoder.md) | class |
| [`Encoded`](./Encoded.md) | class |
| [`Model`](./Model.md) | class |
| [`Normalizer`](./Normalizer.md) | class |
| [`PostProcessor`](./PostProcessor.md) | class |
| [`PreTokenizer`](./PreTokenizer.md) | class |
| [`StreamingDecoder`](./StreamingDecoder.md) | class |
| [`Tokenizer`](./Tokenizer.md) | class |
| [`TokenizerBuilder`](./TokenizerBuilder.md) | class |
| [functions](./functions.md) | module functions |