Skip to main content

ClikaRT::tokenizer::StreamingDecoder

class

Header: ClikaRT/tokenizer/tokenizer.h

Incremental id→text decoding for a generation stream. Ids arrive one at a time, but decoded bytes are only displayable at UTF-8 boundaries (a code point can span tokens); push returns exactly the newly-stable text (often empty while a code point is incomplete) and finish flushes the tail and resets the decoder for a fresh stream. Obtain one from Tokenizer::streaming_decoder; the handle shares the tokenizer's internals and stays valid even after that Tokenizer is destroyed. Move-only.

Member functions​

StreamingDecoder()​

StreamingDecoder(StreamingDecoder&&) noexcept

Move-only: transfers the decoder state.

Declared in ClikaRT/tokenizer/tokenizer.h, line 156

operator=()​

StreamingDecoder& operator=(StreamingDecoder&&) noexcept

Move-assign: transfers the decoder state.

Declared in ClikaRT/tokenizer/tokenizer.h, line 158

~StreamingDecoder()​

~StreamingDecoder()

Releases the decoder (the shared internals outlive the source Tokenizer).

Declared in ClikaRT/tokenizer/tokenizer.h, line 160

push()​

std::string push(std::int32_t id)

Append one generated id; returns the newly-stable text.

Declared in ClikaRT/tokenizer/tokenizer.h, line 164

finish()​

std::string finish()

Flush everything decoded but not yet emitted (including a trailing incomplete sequence as-is); the decoder resets, reusable for a fresh stream.

Declared in ClikaRT/tokenizer/tokenizer.h, line 170