Skip to main content

Tokenize text and apply a chat template

Your model consumes token ids, and a chat model expects its prompt formatted exactly the way it was trained. ClikaRT::Tokenizer covers both: one loader reads a HuggingFace model directory, encode/decode convert text to ids and back, and the model's own chat template renders conversations. No Python and no external tokenizer library are involved.

The programs below use the tokenizer of SmolLM2-135M-Instruct, the model from the GGUF guide. Two small files are all a tokenizer needs:

mkdir -p SmolLM2-135M-Instruct && cd SmolLM2-135M-Instruct
curl -LO "https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct/resolve/main/tokenizer.json"
curl -LO "https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct/resolve/main/tokenizer_config.json"
cd ..

The bundle also ships a self-contained tokenizer under examples/src/tokenizer/data/hf_model, if you would rather not download anything. The C samples abbreviate the api-table bootstrap that tutorial part 1 shows in full.

Load a tokenizer and round-trip some text​

Tokenizer::from_huggingface takes the model directory, detects the artifact inside it (tokenizer.json, tokenizer.model, tekken.json, or vocab.json plus merges.txt), and overlays tokenizer_config.json for the special-token ids and the chat template. Tokenizer::from_file loads one tokenizer file (or a directory holding one) directly, detecting its format the same way.

roundtrip.cpp
#include <cstdint>
#include <cstdio>
#include <string>
#include <vector>

#include "ClikaRT/clika_rt.h"

using ClikaRT::tokenizer::Tokenizer;

int main() {
    Tokenizer tok = Tokenizer::from_huggingface("SmolLM2-135M-Instruct");

    std::printf("vocab %lld, bos %lld, eos %lld, chat template: %s\n",
                (long long)tok.vocab_size(), (long long)tok.bos_id(),
                (long long)tok.eos_id(), tok.has_chat_template() ? "yes" : "no");

    const std::string text = "ClikaRT runs the same code on every backend.";
    const std::vector<std::int32_t> ids = tok.encode(text);
    std::printf("encoded %zu tokens:", ids.size());
    for (std::int32_t id : ids) std::printf(" %d", id);
    std::printf("\ndecoded: %s\n", tok.decode(ids).c_str());
    return 0;
}
vocab 49152, bos 1, eos 2, chat template: yes
encoded 12 tokens: 51 1418 6335 16895 7313 260 1142 2909 335 897 25817 30
decoded: ClikaRT runs the same code on every backend.

See where each token came from​

tokenize returns one Token per piece: the id, the surface string, and the byte span [begin, end) in the original text. Slice the original by that span when you need alignment (highlighting, span labeling, streaming cursors); the spans line up exactly, dropped spaces included.

offsets.cpp
#include <cstdio>
#include <string>
#include <string_view>
#include <vector>

#include "ClikaRT/clika_rt.h"

using ClikaRT::tokenizer::Token;
using ClikaRT::tokenizer::Tokenizer;

int main() {
    Tokenizer tok = Tokenizer::from_huggingface("SmolLM2-135M-Instruct");

    const std::string text = "Quantized weights stay packed.";
    const std::vector<Token> tokens = tok.tokenize(text, /*add_special_tokens=*/false);

    std::printf("  id      [begin,end)  source span\n");
    for (const Token& t : tokens) {
        const std::string_view span(text.data() + t.begin, t.end - t.begin);
        std::printf("  %-6d  [%2zu,%2zu)      \"%.*s\"\n",
                    t.id, t.begin, t.end, (int)span.size(), span.data());
    }
    return 0;
}
  id      [begin,end)  source span
  24696   [ 0, 5)      "Quant"
  1005    [ 5, 9)      "ized"
  10379   [ 9,17)      " weights"
  2951    [17,22)      " stay"
  13448   [22,29)      " packed"
  30      [29,30)      "."

Render a conversation with the model's chat template​

A chat model's prompt format (its role markers, turn separators, generation priming) ships with the model as a Jinja2 template in tokenizer_config.json, and the loader attached it above. apply_chat_template renders a messages array the OpenAI-API shape into the exact prompt string; encode_chat goes straight to ids. Never hand-build these markers: the template is the model's contract.

chat_template.cpp
#include <cstdint>
#include <cstdio>
#include <string>
#include <vector>

#include "ClikaRT/clika_rt.h"

using ClikaRT::json::Json;
using ClikaRT::tokenizer::Tokenizer;

int main() {
    Tokenizer tok = Tokenizer::from_huggingface("SmolLM2-135M-Instruct");

    Json messages = Json::array();
    Json system = Json::object();
    system["role"] = "system";
    system["content"] = "You are a concise assistant.";
    messages.push_back(std::move(system));
    Json user = Json::object();
    user["role"] = "user";
    user["content"] = "What does a tokenizer do?";
    messages.push_back(std::move(user));

    const std::string prompt = tok.apply_chat_template(messages);
    std::printf("=== rendered prompt ===\n%s\n=======================\n", prompt.c_str());

    const std::vector<std::int32_t> ids =
        tok.encode_chat(messages, /*add_generation_prompt=*/true,
                        /*add_special_tokens=*/false);
    std::printf("encode_chat produced %zu tokens\n", ids.size());
    return 0;
}
=== rendered prompt ===
<|im_start|>system
You are a concise assistant.<|im_end|>
<|im_start|>user
What does a tokenizer do?<|im_end|>
<|im_start|>assistant

=======================
encode_chat produced 26 tokens

The rendered prompt ends with the assistant-turn priming (add_generation_prompt defaults to true), so the model continues as the assistant. For tool calling, extra template variables, or a reproducible clock, pass a ChatTemplateInputs instead of the bare messages array; the two-argument form above covers plain conversations.

Batch for a model​

Feeding a model takes tensors, not vectors. encode_batch with return_tensors produces the standard quartet: padded input_ids [B, S], an attention_mask, per-sequence lengths, and the cu_seqlens prefix-sum table. varlen = true skips padding entirely and lays the ids out flat, the shape variable-length attention consumes.

batch.cpp
#include <cstdio>
#include <string_view>
#include <vector>

#include "ClikaRT/clika_rt.h"

using ClikaRT::tokenizer::EncodeOptions;
using ClikaRT::tokenizer::Tokenizer;

int main() {
    Tokenizer tok = Tokenizer::from_huggingface("SmolLM2-135M-Instruct");

    const std::vector<std::string_view> texts = {
        "Short prompt.",
        "A somewhat longer prompt that pads the short one.",
    };

    EncodeOptions opts;
    opts.return_tensors = true;
    // Decoder-only checkpoints often ship no pad token; designate one (eos is
    // the usual choice) or the padded encode raises ClikaRT::Error.
    opts.pad_id = static_cast<std::int32_t>(tok.eos_id());
    const ClikaRT::tokenizer::Encoded batch = tok.encode_batch(texts, opts);
    std::printf("input_ids      %s\n", batch.input_ids->to_string().c_str());
    std::printf("attention_mask %s\n", batch.attention_mask->to_string().c_str());
    std::printf("seq_lengths    %s\n", batch.seq_lengths->to_string().c_str());

    opts.varlen = true;
    const ClikaRT::tokenizer::Encoded flat = tok.encode_batch(texts, opts);
    std::printf("varlen ids     %s\n", flat.input_ids->to_string().c_str());
    std::printf("cu_seqlens     %s\n", flat.cu_seqlens->to_string().c_str());
    return 0;
}
input_ids      Tensor(shape=[2, 10], dtype=Int32, device=CPU, numel=20, data=[20355, 6011, 30, 2, 2, 2, ...])
attention_mask Tensor(shape=[2, 10], dtype=Int32, device=CPU, numel=20, data=[1, 1, 1, 0, 0, 0, ...])
seq_lengths    Tensor(shape=[2], dtype=Int32, device=CPU, numel=2, data=[3, 10])
varlen ids     Tensor(shape=[13], dtype=Int32, device=CPU, numel=13, data=[20355, 6011, 30, 49, 7932, 2848, ...])
cu_seqlens     Tensor(shape=[3], dtype=Int32, device=CPU, numel=3, data=[0, 3, 13])

EncodeOptions also carries truncation (max_length, truncation_side), the padding side (Left suits decoder-only batch generation), and a target device so the tensors land where the model computes. The bundle's tokenizer example walks each of these one chapter at a time, and the templating example covers the Jinja2-compatible engine behind apply_chat_template on its own.

Stream the decode of a generation loop​

A generation loop produces ids one at a time, and decode(ids) over the growing list re-decodes everything on every step. Tokenizer::streaming_decoder is the incremental form: push(id) returns exactly the newly-stable text, and the pieces concatenate to what decode would have produced. The catch it handles for you is the UTF-8 boundary: one code point can span tokens, so push holds bytes back until they are displayable and returns an empty string meanwhile; you never emit half a character. finish() flushes whatever the tail held (a trailing incomplete sequence as-is) and resets the decoder for a fresh stream; a decoder serves one stream at a time.

stream_decode.cpp
#include <cstdint>
#include <cstdio>
#include <string>
#include <vector>

#include "ClikaRT/clika_rt.h"

using ClikaRT::tokenizer::StreamingDecoder;
using ClikaRT::tokenizer::Tokenizer;

int main() {
    Tokenizer tok = Tokenizer::from_huggingface("SmolLM2-135M-Instruct");

    // Stand-in for a generation loop: the ids a real decoder would emit one
    // at a time (the roundtrip section's sentence, so the ids match).
    const std::vector<std::int32_t> ids =
        tok.encode("ClikaRT runs the same code on every backend.");

    StreamingDecoder stream = tok.streaming_decoder();
    std::string assembled;
    int emitted = 0;
    for (std::int32_t id : ids) {
        // push returns exactly the newly-stable text: empty while a
        // multi-byte code point is still incomplete, never a torn character.
        const std::string piece = stream.push(id);
        if (!piece.empty()) ++emitted;
        assembled += piece;
    }
    assembled += stream.finish();  // flush the tail; the decoder resets

    std::printf("%zu ids -> %d incremental pieces\n", ids.size(), emitted);
    std::printf("assembled: %s\n", assembled.c_str());
    std::printf("assembled == decode(ids): %s\n",
                assembled == tok.decode(ids) ? "yes" : "no");
    return 0;
}
12 ids -> 12 incremental pieces
assembled: ClikaRT runs the same code on every backend.
assembled == decode(ids): yes

Every push emitted text here because the sentence is plain ASCII; text with accents, CJK, or emoji is where the empty returns appear, and exactly why the boundary handling exists. The decoder skips special tokens by default (streaming_decoder(false) keeps them), and the handle stays valid even after the Tokenizer that made it is gone, so a generation worker can own just the decoder.

This is the producer half of token streaming: each non-empty piece is one frame for the transport. The serving guide sends exactly these pieces as token events over server-sent events.