Skip to main content

ClikaRT::tokenizer::PostProcessor

class

Header: ClikaRT/tokenizer/builder.h

The special-token / template stage (optional): wraps encoded sequences with special tokens ([CLS] A [SEP]-style templates, byte-level offset re-encoding). Move-only handle consumed by TokenizerBuilder::post_processor.

Nested types​

NameDescription
SpecialTokenA special token referenced by a template: its key maps to a parallel (tokens, ids) pair.
TemplatePieceOne piece of a processing template: a whole sequence, or a named special.

Static member functions​

byte_level()​

static PostProcessor byte_level(bool add_prefix_space = true, bool trim_offsets = true)

Re-encode byte-level offsets after the model (GPT-2 / RoBERTa family).

Parameters

  • add_prefix_space: match a byte-level pre-tokenizer that inserted a leading space.
  • trim_offsets: trim whitespace out of the reported offsets.

Declared in ClikaRT/tokenizer/builder.h, line 294

roberta()​

static PostProcessor roberta(
    std::string_view sep_token,
    std::int32_t sep_id,
    std::string_view cls_token,
    std::int32_t cls_id,
    bool trim_offsets = true,
    bool add_prefix_space = true
)

RoBERTa <s> ... </s> wrapping.

Parameters

  • sep_token: the separator token text.
  • sep_id: the separator token's vocabulary id.
  • cls_token: the classifier token text.
  • cls_id: the classifier token's vocabulary id.
  • trim_offsets: trim whitespace out of reported offsets.
  • add_prefix_space: match a prefix-space byte-level stage.

Declared in ClikaRT/tokenizer/builder.h, line 302

sequence()​

static PostProcessor sequence(std::vector<PostProcessor> processors)

Apply a list of post-processors in order (each consumed).

Declared in ClikaRT/tokenizer/builder.h, line 306

template_processing()​

static PostProcessor template_processing(
    Span<const TemplatePiece> single,
    Span<const TemplatePiece> pair,
    Span<const SpecialToken> special_tokens
)

Assemble sequences from a single/pair template referencing the given special tokens (the BERT [CLS] A [SEP] / [CLS] A [SEP] B [SEP] shape).

Declared in ClikaRT/tokenizer/builder.h, line 309

Member functions​

PostProcessor(void)​

explicit PostProcessor(void* impl) noexcept

Declared in ClikaRT/tokenizer/builder.h, line 313

PostProcessor(PostProcessor)​

PostProcessor(PostProcessor&& other) noexcept

Move-only handle; the moved-from object is empty.

Declared in ClikaRT/tokenizer/builder.h, line 315

operator=()​

PostProcessor& operator=(PostProcessor&& other) noexcept

Move-only handle; the moved-from object is empty.

Declared in ClikaRT/tokenizer/builder.h, line 317

~PostProcessor()​

~PostProcessor()

Releases the post-processor.

Declared in ClikaRT/tokenizer/builder.h, line 319