Skip to main content

ClikaRT::tokenizer::Encoded

struct

Header: ClikaRT/tokenizer/tokenizer.h

Tensor-shaped encode output. ids always carries the per-sequence id rows (one inner vector per input text). The tensor columns are present only when EncodeOptions::return_tensors is set:

  • input_ids: [B, S] padded Int32 (per EncodeOptions::padding_side, right by default) or [ΣS] flat Int32 (varlen).
  • attention_mask: [B, S] Int32, 1 for a token / 0 for pad (padded path only).
  • seq_lengths: [B] Int32, the per-sequence token counts.
  • cu_seqlens: [B + 1] Int32, the exclusive prefix-sum offset table, the varlen attention contract's index dtype, so it feeds attention directly.

Data members​

ids​

std::vector<std::vector<std::int32_t>> ids

per-sequence id rows; always present

Declared in ClikaRT/tokenizer/tokenizer.h, line 81

input_ids​

std::optional<Tensor> input_ids

[B, S] padded or [SumS] varlen Int32 (see the struct block)

Declared in ClikaRT/tokenizer/tokenizer.h, line 82

attention_mask​

std::optional<Tensor> attention_mask

[B, S] Int32, 1 = token / 0 = pad (padded path only)

Declared in ClikaRT/tokenizer/tokenizer.h, line 83

seq_lengths​

std::optional<Tensor> seq_lengths

[B] Int32 per-sequence token counts

Declared in ClikaRT/tokenizer/tokenizer.h, line 84

cu_seqlens​

std::optional<Tensor> cu_seqlens

[B + 1] Int32 exclusive prefix sums (varlen attention's table)

Declared in ClikaRT/tokenizer/tokenizer.h, line 85