Skip to main content

ClikaRT::tokenizer::TokenizerBuilder

class

Header: ClikaRT/tokenizer/builder.h

Assembles a Tokenizer from a required Model plus the optional pipeline stages.

Fluent: every setter consumes its stage handle and returns *this, so a build reads as one chain ending in build(), which validates the assembled configuration (template/special-token consistency, the chat template compiles) and raises ClikaRT::Error on an inconsistent one.

auto tok = ClikaRT::TokenizerBuilder(std::move(model))
               .normalizer(ClikaRT::Normalizer::nfc())
               .pre_tokenizer(ClikaRT::PreTokenizer::byte_level())
               .eos_id(2)
               .build();

Member functions​

TokenizerBuilder(Model, StreamOrDevice)​

explicit TokenizerBuilder(Model model, StreamOrDevice where = {})

Start from the required model. where is the default placement for the tokenizer's tensor-shaped entry points.

Declared in ClikaRT/tokenizer/builder.h, line 396

normalizer()​

TokenizerBuilder& normalizer(Normalizer normalizer)

Set a pipeline stage (each consumes its handle). Chainable.

Declared in ClikaRT/tokenizer/builder.h, line 399

pre_tokenizer()​

TokenizerBuilder& pre_tokenizer(PreTokenizer pre_tokenizer)

Set the pre-segmentation stage (consumes the handle). Chainable.

Declared in ClikaRT/tokenizer/builder.h, line 402

post_processor()​

TokenizerBuilder& post_processor(PostProcessor post_processor)

Set the special-token / template stage (consumes the handle). Chainable.

Declared in ClikaRT/tokenizer/builder.h, line 405

decoder()​

TokenizerBuilder& decoder(Decoder decoder)

Set the detokenization stage (consumes the handle). Chainable.

Declared in ClikaRT/tokenizer/builder.h, line 408

chat_template()​

TokenizerBuilder& chat_template(
    std::string_view jinja_source,
    std::string_view bos_token,
    std::string_view eos_token
)

Attach a chat template (a Jinja source string) so Tokenizer's apply_chat_template / encode_chat can render chat turns. bos_token / eos_token are the framing tokens the template references. The source is compiled at build(); an invalid template surfaces as a build() error.

Declared in ClikaRT/tokenizer/builder.h, line 414

add_token()​

TokenizerBuilder& add_token(
    std::string_view content,
    std::int32_t id,
    bool single_word = false,
    bool lstrip = false,
    bool rstrip = false,
    bool normalized = true
)

Register an added token: an ordinary added token (add_token) or a special one (add_special_token, maskable / skippable on decode). Register an ordinary added token.

Parameters

  • content: the token text.
  • id: its id.
  • single_word: match only at word boundaries.
  • lstrip: consume whitespace to the token's left.
  • rstrip: consume whitespace to the token's right.
  • normalized: match against normalized (vs raw) text.

Declared in ClikaRT/tokenizer/builder.h, line 426

add_special_token()​

TokenizerBuilder& add_special_token(std::string_view content, std::int32_t id)

Register the SPECIAL added token content under id (special tokens are maskable / skippable on decode).

Declared in ClikaRT/tokenizer/builder.h, line 431

pad_id()​

TokenizerBuilder& pad_id(std::int32_t id)

Well-known token ids surfaced by the tokenizer (defaults: none / -1).

Declared in ClikaRT/tokenizer/builder.h, line 434

bos_id()​

TokenizerBuilder& bos_id(std::int32_t id)

Set the beginning-of-sequence token id. Chainable.

Declared in ClikaRT/tokenizer/builder.h, line 437

eos_id()​

TokenizerBuilder& eos_id(std::int32_t id)

Set the end-of-sequence token id. Chainable.

Declared in ClikaRT/tokenizer/builder.h, line 439

build()​

Tokenizer build()

Assemble the tokenizer; raises on an inconsistent configuration.

Declared in ClikaRT/tokenizer/builder.h, line 443

TokenizerBuilder(TokenizerBuilder)​

TokenizerBuilder(TokenizerBuilder&& other) noexcept

Declared in ClikaRT/tokenizer/builder.h, line 445

operator=()​

TokenizerBuilder& operator=(TokenizerBuilder&& other) noexcept

Move-only handle; the moved-from object is empty.

Declared in ClikaRT/tokenizer/builder.h, line 447

~TokenizerBuilder()​

~TokenizerBuilder()

Releases the builder (an unbuilt one drops its stages).

Declared in ClikaRT/tokenizer/builder.h, line 450