---
title: "ClikaRT::tokenizer::EncodeOptions"
sidebar_label: "EncodeOptions"
description: "Per-call controls for the tensor-shaped encode entry points."
---

<!-- Generated by tools/doxygen2mdx from Doxygen XML. Do not edit. -->

<span class="api-kind api-kind--struct">struct</span>

**Header:** `ClikaRT/tokenizer/tokenizer.h`

Per-call controls for the tensor-shaped encode entry points.

## Data members

### add\_special\_tokens {#add_special_tokens}

<div class="api-signature">

`bool` **`add_special_tokens`** `= true`

</div>

Wrap the produced ids with the model's special tokens (bos/eos/...).

<div class="api-declared">

Declared in `ClikaRT/tokenizer/tokenizer.h`, line 44

</div>

### return\_tensors {#return_tensors}

<div class="api-signature">

`bool` **`return_tensors`** `= false`

</div>

Build the [`Encoded`](./Encoded.md) tensor columns in addition to `ids`.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/tokenizer.h`, line 46

</div>

### varlen {#varlen}

<div class="api-signature">

`bool` **`varlen`** `= false`

</div>

Lay `input_ids` out as a flat `[ΣS]` ragged buffer with a `cu_seqlens` offset table instead of a padded `[B, S]` matrix. Varlen never pads.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/tokenizer.h`, line 49

</div>

### pad\_id {#pad_id}

<div class="api-signature">

`std::optional<std::int32_t>` **`pad_id`** `= std::nullopt`

</div>

Pad id for a padded batch; falls back to the tokenizer's default pad id.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/tokenizer.h`, line 51

</div>

### device {#device}

<div class="api-signature">

[`StreamOrDevice`](../StreamOrDevice.md) **`device`** `= {}`

</div>

Placement for the produced tensors: a [`Device`](../Device.md), or a [`Stream`](../Stream.md) so the tensors land stream-ordered on that stream (ready to feed a forward on the same stream with no cross-stream wait). Unspecified falls back to the tokenizer's default placement (CPU unless configured otherwise).

<div class="api-declared">

Declared in `ClikaRT/tokenizer/tokenizer.h`, line 56

</div>

### max\_length {#max_length}

<div class="api-signature">

`std::optional<std::int64_t>` **`max_length`** `= std::nullopt`

</div>

Cap each sequence at this many tokens, truncating per [`truncation_side`](./EncodeOptions.md#truncation_side). The model's special tokens are never cut: the core text is trimmed to (max\_length − the specials' count) BEFORE bos/eos attach, so the total stays ≤ max\_length. Unset = no truncation.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/tokenizer.h`, line 61

</div>

### truncation\_side {#truncation_side}

<div class="api-signature">

[`TruncationSide`](./index.md#TruncationSide) **`truncation_side`** `=` [`TruncationSide::Right`](./index.md#TruncationSide)

</div>

which end is trimmed when max\_length binds

<div class="api-declared">

Declared in `ClikaRT/tokenizer/tokenizer.h`, line 62

</div>

### padding\_side {#padding_side}

<div class="api-signature">

[`PaddingSide`](./index.md#PaddingSide) **`padding_side`** `=` [`PaddingSide::Right`](./index.md#PaddingSide)

</div>

Which end a ragged padded batch fills. `seq_lengths` / `cu_seqlens` always carry the REAL lengths either way.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/tokenizer.h`, line 65

</div>

### pad\_to\_multiple\_of {#pad_to_multiple_of}

<div class="api-signature">

`std::optional<std::int64_t>` **`pad_to_multiple_of`** `= std::nullopt`

</div>

Round the padded row length `S` up to a multiple of this (tensor-core friendly shapes). Padded `[B, S]` layout only (an error with [`varlen`](./EncodeOptions.md#varlen)).

<div class="api-declared">

Declared in `ClikaRT/tokenizer/tokenizer.h`, line 68

</div>