---
title: Tokenize text and apply a chat template
description: Turn text into token ids and back, align tokens to source bytes, render a conversation with the model's own chat template, and batch for a model.
---
Your model consumes token ids, and a chat model expects its prompt formatted exactly the way it was trained. `ClikaRT::Tokenizer` covers both: one loader reads a HuggingFace model directory, `encode`/`decode` convert text to ids and back, and the model's own chat template renders conversations. No Python and no external tokenizer library are involved.
The programs below use the tokenizer of [SmolLM2-135M-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct), the model from [the GGUF guide](load-quantized-weights.mdx). Two small files are all a tokenizer needs:
```bash
mkdir -p SmolLM2-135M-Instruct && cd SmolLM2-135M-Instruct
curl -LO "https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct/resolve/main/tokenizer.json"
curl -LO "https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct/resolve/main/tokenizer_config.json"
cd ..
```
The bundle also ships a self-contained tokenizer under `examples/src/tokenizer/data/hf_model`, if you would rather not download anything.
## Load a tokenizer and round-trip some text
`Tokenizer::from_huggingface` takes the model directory, detects the artifact inside it (`tokenizer.json`, `tokenizer.model`, `tekken.json`, or `vocab.json` plus `merges.txt`), and overlays `tokenizer_config.json` for the special-token ids and the chat template. `Tokenizer::from_file` loads one tokenizer file (or a directory holding one) directly, detecting its format the same way.
```cpp title="roundtrip.cpp"
int main() {
Tokenizer tok = Tokenizer::from_huggingface(model_dir());
std::printf("vocab %lld, bos %lld, eos %lld, chat template: %s\n",
static_cast(tok.vocab_size()), static_cast(tok.bos_id()),
static_cast(tok.eos_id()), tok.has_chat_template() ? "yes" : "no");
const std::string text = "ClikaRT runs the same code on every backend.";
const std::vector ids = tok.encode(text);
std::printf("encoded %zu tokens:", ids.size());
for (std::int32_t id : ids) std::printf(" %d", id);
std::printf("\ndecoded: %s\n", tok.decode(ids).c_str());
return 0;
}
```
```python title="roundtrip.py"
def main() -> None:
tok = crt.tokenizer.Tokenizer.from_huggingface(model_dir())
print(f"vocab {tok.vocab_size}, bos {tok.bos_id}, eos {tok.eos_id}, "
f"chat template: {'yes' if tok.has_chat_template else 'no'}")
text = "ClikaRT runs the same code on every backend."
ids = tok.encode(text)
print(f"encoded {len(ids)} tokens:", *ids)
print(f"decoded: {tok.decode(ids)}")
if __name__ == "__main__":
main()
```
```text
vocab 49152, bos 1, eos 2, chat template: yes
encoded 12 tokens: 51 1418 6335 16895 7313 260 1142 2909 335 897 25817 30
decoded: ClikaRT runs the same code on every backend.
```
## See where each token came from
`tokenize` returns one `Token` per piece: the id, the surface string, and the byte span `[begin, end)` in the original text. Slice the original by that span when you need alignment (highlighting, span labeling, streaming cursors); the spans line up exactly, dropped spaces included.
```cpp title="offsets.cpp"
int main() {
Tokenizer tok = Tokenizer::from_huggingface(model_dir());
const std::string text = "Quantized weights stay packed.";
const std::vector tokens = tok.tokenize(text, /*add_special_tokens=*/false);
std::printf(" id [begin,end) source span\n");
for (const Token& t : tokens) {
const std::string_view span(text.data() + t.begin, t.end - t.begin);
std::printf(" %-6d [%2zu,%2zu) \"%.*s\"\n",
t.id, t.begin, t.end, static_cast(span.size()), span.data());
}
return 0;
}
```
```python title="offsets.py"
def main() -> None:
tok = crt.tokenizer.Tokenizer.from_huggingface(model_dir())
text = "Quantized weights stay packed."
ids = tok.encode(text, add_special_tokens=False)
# The python binding returns ids; id_to_token shows each piece. A
# leading 'G-with-breve' marks a token that starts with a space; the
# byte-span view is the C++ tokenize surface.
print(" id token")
for i in ids:
print(f" {i:<6} {tok.id_to_token(i)!r}")
if __name__ == "__main__":
main()
```
```text
id [begin,end) source span
24696 [ 0, 5) "Quant"
1005 [ 5, 9) "ized"
10379 [ 9,17) " weights"
2951 [17,22) " stay"
13448 [22,29) " packed"
30 [29,30) "."
```
## Render a conversation with the model's chat template
A chat model's prompt format (its role markers, turn separators, generation priming) ships with the model as a Jinja2 template in `tokenizer_config.json`, and the loader attached it above. `apply_chat_template` renders a `messages` array the OpenAI-API shape into the exact prompt string; `encode_chat` goes straight to ids. Never hand-build these markers: the template is the model's contract.
```cpp title="chat_template.cpp"
int main() {
Tokenizer tok = Tokenizer::from_huggingface(model_dir());
Json messages = Json::array();
Json system = Json::object();
system["role"] = "system";
system["content"] = "You are a concise assistant.";
messages.push_back(std::move(system));
Json user = Json::object();
user["role"] = "user";
user["content"] = "What does a tokenizer do?";
messages.push_back(std::move(user));
const std::string prompt = tok.apply_chat_template(messages);
std::printf("=== rendered prompt ===\n%s\n=======================\n", prompt.c_str());
// The template writes the prompt's own bos/eos framing, so encode_chat adds
// no special tokens on top of it.
const std::vector ids = tok.encode_chat(messages, /*add_generation_prompt=*/true);
std::printf("encode_chat produced %zu tokens\n", ids.size());
return 0;
}
```
```python title="chat_template.py"
def main() -> None:
tok = crt.tokenizer.Tokenizer.from_huggingface(model_dir())
# The messages array travels as JSON text at this surface.
messages = json.dumps([
{"role": "system", "content": "You are a concise assistant."},
{"role": "user", "content": "What does a tokenizer do?"},
])
prompt = tok.apply_chat_template(messages)
print(f"=== rendered prompt ===\n{prompt}\n=======================")
# The template writes the prompt's own bos/eos framing, so encode_chat adds
# no special tokens on top of it.
ids = tok.encode_chat(messages, add_generation_prompt=True)
print(f"encode_chat produced {len(ids)} tokens")
if __name__ == "__main__":
main()
```
```text
=== rendered prompt ===
<|im_start|>system
You are a concise assistant.<|im_end|>
<|im_start|>user
What does a tokenizer do?<|im_end|>
<|im_start|>assistant
=======================
encode_chat produced 107 tokens
```
The rendered prompt ends with the assistant-turn priming (`add_generation_prompt` defaults to true), so the model continues as the assistant. For tool calling, extra template variables, or a reproducible clock, pass a `ChatTemplateInputs` instead of the bare messages array; the two-argument form above covers plain conversations.
## Batch for a model
Feeding a model takes tensors, not vectors. `encode_batch` with `return_tensors` produces the standard quartet: padded `input_ids` `[B, S]`, an `attention_mask`, per-sequence lengths, and the `cu_seqlens` prefix-sum table. `varlen = true` skips padding entirely and lays the ids out flat, the shape variable-length attention consumes.
```cpp title="batch.cpp"
int main() {
Tokenizer tok = Tokenizer::from_huggingface(model_dir());
const std::vector texts = {
"Short prompt.",
"A somewhat longer prompt that pads the short one.",
};
EncodeOptions opts;
opts.return_tensors = true;
// Decoder-only checkpoints often ship no pad token; designate one (eos is
// the usual choice) or the padded encode raises ClikaRT::Error.
opts.pad_id = static_cast(tok.eos_id());
const ClikaRT::tokenizer::Encoded batch = tok.encode_batch(texts, opts);
std::printf("input_ids %s\n", batch.input_ids->to_string().c_str());
std::printf("attention_mask %s\n", batch.attention_mask->to_string().c_str());
std::printf("seq_lengths %s\n", batch.seq_lengths->to_string().c_str());
opts.varlen = true;
const ClikaRT::tokenizer::Encoded flat = tok.encode_batch(texts, opts);
std::printf("varlen ids %s\n", flat.input_ids->to_string().c_str());
std::printf("cu_seqlens %s\n", flat.cu_seqlens->to_string().c_str());
return 0;
}
```
```python title="batch.py"
def main() -> None:
tok = crt.tokenizer.Tokenizer.from_huggingface(model_dir())
texts = [
"Short prompt.",
"A somewhat longer prompt that pads the short one.",
]
# The python binding returns the ragged ids, one list per text; pad on
# the tensor side with the lengths below. The padded quartet (input_ids,
# attention_mask, seq_lengths, cu_seqlens) is the C++ and C surface.
batch = tok.encode_batch(texts)
for row, ids in enumerate(batch.ids):
print(f"text {row}: {len(ids):2} ids {ids}")
if __name__ == "__main__":
main()
```
```text
input_ids Tensor(shape=[2, 10], dtype=Int32, device=CPU, numel=20, data=[20355, 6011, 30, 2, 2, 2, ...])
attention_mask Tensor(shape=[2, 10], dtype=Int32, device=CPU, numel=20, data=[1, 1, 1, 0, 0, 0, ...])
seq_lengths Tensor(shape=[2], dtype=Int32, device=CPU, numel=2, data=[3, 10])
varlen ids Tensor(shape=[13], dtype=Int32, device=CPU, numel=13, data=[20355, 6011, 30, 49, 7932, 2848, ...])
cu_seqlens Tensor(shape=[3], dtype=Int32, device=CPU, numel=3, data=[0, 3, 13])
```
`EncodeOptions` also carries truncation (`max_length`, `truncation_side`), the padding side (`Left` suits decoder-only batch generation), and a target `device` so the tensors land where the model computes. The bundle's `tokenizer` example walks each of these one chapter at a time, and the `templating` example covers the Jinja2-compatible engine behind `apply_chat_template` on its own.
## Stream the decode of a generation loop
A generation loop produces ids one at a time, and `decode(ids)` over the growing list re-decodes everything on every step. `Tokenizer::streaming_decoder` is the incremental form: `push(id)` returns exactly the newly-stable text, and the pieces concatenate to what `decode` would have produced. The catch it handles for you is the UTF-8 boundary: one code point can span tokens, so `push` holds bytes back until they are displayable and returns an empty string meanwhile; you never emit half a character. `finish()` flushes whatever the tail held (a trailing incomplete sequence as-is) and resets the decoder for a fresh stream; a decoder serves one stream at a time.
```cpp title="stream_decode.cpp"
int main() {
Tokenizer tok = Tokenizer::from_huggingface(model_dir());
// Stand-in for a generation loop: the ids a real decoder would emit one
// at a time (the roundtrip section's sentence, so the ids match).
const std::vector ids =
tok.encode("ClikaRT runs the same code on every backend.");
StreamingDecoder stream = tok.streaming_decoder();
std::string assembled;
int emitted = 0;
for (std::int32_t id : ids) {
// push returns exactly the newly-stable text: empty while a
// multi-byte code point is still incomplete, never a torn character.
const std::string piece = stream.push(id);
if (!piece.empty()) ++emitted;
assembled += piece;
}
assembled += stream.finish(); // flush the tail; the decoder resets
std::printf("%zu ids -> %d incremental pieces\n", ids.size(), emitted);
std::printf("assembled: %s\n", assembled.c_str());
std::printf("assembled == decode(ids): %s\n",
assembled == tok.decode(ids) ? "yes" : "no");
return 0;
}
```
```python title="stream_decode.py"
def main() -> None:
tok = crt.tokenizer.Tokenizer.from_huggingface(model_dir())
# Stand-in for a generation loop: the ids a real decoder would emit one
# at a time (the roundtrip section's sentence, so the ids match).
ids = tok.encode("ClikaRT runs the same code on every backend.")
stream = tok.streaming_decoder()
pieces = [stream.push(i) for i in ids] # "" while a code point is incomplete
assembled = "".join(pieces) + stream.finish() # flush; the decoder resets
print(f"{len(ids)} ids -> {sum(1 for p in pieces if p)} incremental pieces")
print(f"assembled: {assembled}")
print(f"assembled == decode(ids): "
f"{'yes' if assembled == tok.decode(ids) else 'no'}")
if __name__ == "__main__":
main()
```
{/* CERTIFICATION: the run below follows the API contract (the pieces concatenate to decode(ids));
the piece count is not verified against a built bundle. */}
```text
32 ids -> 32 incremental pieces
assembled: ClikaRT runs the same code on every backend.
assembled == decode(ids): yes
```
Every push emitted text here because the sentence is plain ASCII; text with accents, CJK, or emoji is where the empty returns appear, and exactly why the boundary handling exists. The decoder skips special tokens by default (`streaming_decoder(false)` keeps them), and the handle stays valid even after the `Tokenizer` that made it is gone, so a generation worker can own just the decoder.
This is the producer half of token streaming: each non-empty piece is one frame for the transport. [The serving guide](serve-over-http.mdx) sends exactly these pieces as `token` events over server-sent events.