---
title: "ClikaRT::tokenizer::StreamingDecoder"
sidebar_label: "StreamingDecoder"
description: "Incremental id→text decoding for a generation stream. Ids arrive one at a time, but decoded bytes are only displayable at UTF-8 boundaries (a code point can span tokens); push returns exactly the newl…"
---

<!-- Generated by tools/doxygen2mdx from Doxygen XML. Do not edit. -->

<span class="api-kind api-kind--class">class</span>

**Header:** `ClikaRT/tokenizer/tokenizer.h`

Incremental id→text decoding for a generation stream. Ids arrive one at a time, but decoded bytes are only displayable at UTF-8 boundaries (a code point can span tokens); [`push`](./StreamingDecoder.md#push) returns exactly the newly-stable text (often empty while a code point is incomplete) and [`finish`](./StreamingDecoder.md#finish) flushes the tail and resets the decoder for a fresh stream. Obtain one from [`Tokenizer::streaming_decoder`](./Tokenizer.md#streaming_decoder); the handle shares the tokenizer's internals and stays valid even after that [`Tokenizer`](./Tokenizer.md) is destroyed. Move-only.

## Member functions

### StreamingDecoder() {#StreamingDecoder}

<div class="api-signature">

**`StreamingDecoder`**(`StreamingDecoder&&`) `noexcept`

</div>

Move-only: transfers the decoder state.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/tokenizer.h`, line 156

</div>

### operator=() {#operator-assign}

<div class="api-signature">

[`StreamingDecoder`](./StreamingDecoder.md)`&` **`operator=`**([`StreamingDecoder`](./StreamingDecoder.md)`&&`) `noexcept`

</div>

Move-assign: transfers the decoder state.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/tokenizer.h`, line 158

</div>

### \~StreamingDecoder() {#StreamingDecoder-2}

<div class="api-signature">

**`~StreamingDecoder`**()

</div>

Releases the decoder (the shared internals outlive the source [`Tokenizer`](./Tokenizer.md)).

<div class="api-declared">

Declared in `ClikaRT/tokenizer/tokenizer.h`, line 160

</div>

### push() {#push}

<div class="api-signature">

`std::string` **`push`**(`std::int32_t` `id`)

</div>

Append one generated id; returns the newly-stable text.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/tokenizer.h`, line 164

</div>

### finish() {#finish}

<div class="api-signature">

`std::string` **`finish`**()

</div>

Flush everything decoded but not yet emitted (including a trailing incomplete sequence as-is); the decoder resets, reusable for a fresh stream.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/tokenizer.h`, line 170

</div>