---
title: "ClikaRT::tokenizer::PostProcessor::SpecialToken"
sidebar_label: "PostProcessor::SpecialToken"
description: "A special token referenced by a template: its key maps to a parallel (tokens, ids) pair."
---

<!-- Generated by tools/doxygen2mdx from Doxygen XML. Do not edit. -->

<span class="api-kind api-kind--struct">struct</span>

**Header:** `ClikaRT/tokenizer/builder.h`

A special token referenced by a template: its [`key`](./PostProcessor.SpecialToken.md#key) maps to a parallel `(`[`tokens`](./PostProcessor.SpecialToken.md#tokens)`, `[`ids`](./PostProcessor.SpecialToken.md#ids)`)` pair.

## Data members

### key {#key}

<div class="api-signature">

`std::string_view` **`key`**

</div>

The key template pieces reference.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 279

</div>

### tokens {#tokens}

<div class="api-signature">

`std::vector<std::string_view>` **`tokens`**

</div>

The token strings emitted for this key. The vector OWNS the list (assigning from a braced list is safe); the referenced STRINGS must stay alive until the build call, like every `string_view` input on this builder.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 284

</div>

### ids {#ids}

<div class="api-signature">

`std::vector<std::int32_t>` **`ids`**

</div>

The ids emitted alongside [`tokens`](./PostProcessor.SpecialToken.md#tokens) (parallel, same length). The vector owns its values.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 287

</div>