---
title: "ClikaRT::tokenizer::PostProcessor"
sidebar_label: "PostProcessor"
description: "The special-token / template stage (optional): wraps encoded sequences with special tokens ([CLS] A [SEP]-style templates, byte-level offset re-encoding)."
---

<!-- Generated by tools/doxygen2mdx from Doxygen XML. Do not edit. -->

<span class="api-kind api-kind--class">class</span>

**Header:** `ClikaRT/tokenizer/builder.h`

The special-token / template stage (optional): wraps encoded sequences with special tokens (`[CLS] A [SEP]`-style templates, byte-level offset re-encoding). Move-only handle consumed by [`TokenizerBuilder::post_processor`](./TokenizerBuilder.md#post_processor).

## Nested types

| Name | Description |
| --- | --- |
| [`SpecialToken`](./PostProcessor.SpecialToken.md) | A special token referenced by a template: its [`key`](./PostProcessor.SpecialToken.md#key) maps to a parallel `(`[`tokens`](./PostProcessor.SpecialToken.md#tokens)`, `[`ids`](./PostProcessor.SpecialToken.md#ids)`)` pair. |
| [`TemplatePiece`](./PostProcessor.TemplatePiece.md) | One piece of a processing template: a whole sequence, or a named special. |

## Static member functions

### byte\_level() {#byte_level}

<div class="api-signature">

`static` [`PostProcessor`](./PostProcessor.md) **`byte_level`**(`bool` `add_prefix_space` `=` `true`, `bool` `trim_offsets` `=` `true`)

</div>

Re-encode byte-level offsets after the model (GPT-2 / RoBERTa family).

**Parameters**

- `add_prefix_space`: match a byte-level pre-tokenizer that inserted a leading space.
- `trim_offsets`: trim whitespace out of the reported offsets.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 294

</div>

### roberta() {#roberta}

<div class="api-signature">

`static` [`PostProcessor`](./PostProcessor.md) **`roberta`**(<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::string_view` `sep_token`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::int32_t` `sep_id`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::string_view` `cls_token`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::int32_t` `cls_id`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`bool` `trim_offsets` `=` `true`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`bool` `add_prefix_space` `=` `true`<br />
)

</div>

RoBERTa `<s> ... </s>` wrapping.

**Parameters**

- `sep_token`: the separator token text.
- `sep_id`: the separator token's vocabulary id.
- `cls_token`: the classifier token text.
- `cls_id`: the classifier token's vocabulary id.
- `trim_offsets`: trim whitespace out of reported offsets.
- `add_prefix_space`: match a prefix-space byte-level stage.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 302

</div>

### sequence() {#sequence}

<div class="api-signature">

`static` [`PostProcessor`](./PostProcessor.md) **`sequence`**(`std::vector<`[`PostProcessor`](./PostProcessor.md)`>` `processors`)

</div>

Apply a list of post-processors in order (each consumed).

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 306

</div>

### template\_processing() {#template_processing}

<div class="api-signature">

`static` [`PostProcessor`](./PostProcessor.md) **`template_processing`**(<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`Span`](../Span.md)`<const` [`TemplatePiece`](./PostProcessor.TemplatePiece.md)`>` `single`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`Span`](../Span.md)`<const` [`TemplatePiece`](./PostProcessor.TemplatePiece.md)`>` `pair`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;[`Span`](../Span.md)`<const` [`SpecialToken`](./PostProcessor.SpecialToken.md)`>` `special_tokens`<br />
)

</div>

Assemble sequences from a `single`/`pair` template referencing the given special tokens (the BERT `[CLS] A [SEP]` / `[CLS] A [SEP] B [SEP]` shape).

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 309

</div>

## Member functions

### PostProcessor(void) {#PostProcessor}

<div class="api-signature">

`explicit` **`PostProcessor`**(`void*` `impl`) `noexcept`

</div>

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 313

</div>

### PostProcessor(PostProcessor) {#PostProcessor-2}

<div class="api-signature">

**`PostProcessor`**(`PostProcessor&&` `other`) `noexcept`

</div>

Move-only handle; the moved-from object is empty.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 315

</div>

### operator=() {#operator-assign}

<div class="api-signature">

[`PostProcessor`](./PostProcessor.md)`&` **`operator=`**([`PostProcessor`](./PostProcessor.md)`&&` `other`) `noexcept`

</div>

Move-only handle; the moved-from object is empty.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 317

</div>

### \~PostProcessor() {#PostProcessor-3}

<div class="api-signature">

**`~PostProcessor`**()

</div>

Releases the post-processor.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 319

</div>