---
title: "ClikaRT::tokenizer::Token"
sidebar_label: "Token"
description: "One token produced by tokenize: its id, the surface string it spells, and the half-open byte span [begin, end) it covers in the source text."
---

<!-- Generated by tools/doxygen2mdx from Doxygen XML. Do not edit. -->

<span class="api-kind api-kind--struct">struct</span>

**Header:** `ClikaRT/tokenizer/tokenizer.h`

One token produced by `tokenize`: its id, the surface string it spells, and the half-open byte span `[`[`begin`](./Token.md#begin)`, `[`end`](./Token.md#end)`)` it covers in the source text. [`special`](./Token.md#special) is true for a token the tokenizer injected itself (bos/eos/...), not one that came from the input.

## Data members

### id {#id}

<div class="api-signature">

`std::int32_t` **`id`** `= 0`

</div>

vocabulary id

<div class="api-declared">

Declared in `ClikaRT/tokenizer/tokenizer.h`, line 139

</div>

### text {#text}

<div class="api-signature">

`std::string` **`text`**

</div>

the surface piece (UTF-8)

<div class="api-declared">

Declared in `ClikaRT/tokenizer/tokenizer.h`, line 140

</div>

### begin {#begin}

<div class="api-signature">

`std::size_t` **`begin`** `= 0`

</div>

byte span start in the source text

<div class="api-declared">

Declared in `ClikaRT/tokenizer/tokenizer.h`, line 141

</div>

### end {#end}

<div class="api-signature">

`std::size_t` **`end`** `= 0`

</div>

byte span end (half-open)

<div class="api-declared">

Declared in `ClikaRT/tokenizer/tokenizer.h`, line 142

</div>

### special {#special}

<div class="api-signature">

`bool` **`special`** `= false`

</div>

injected by the tokenizer (bos/eos/...), not from the input

<div class="api-declared">

Declared in `ClikaRT/tokenizer/tokenizer.h`, line 143

</div>