---
title: "tokenize"
sidebar_label: "tokenize"
description: "Kotlin binding reference: tokenize."
---

<!-- Generated by tools/api_reference/generate_api_docs.py. Do not edit. -->

//[clika-runtime](../../../index.md)/[io.clika.runtime](../index.md)/[Tokenizer](index.md)/[tokenize](tokenize.md)

# tokenize

[common]\
fun [tokenize](tokenize.md)(text: [String](https://kotlinlang.org/api/core/kotlin-stdlib/kotlin/-string/index.html), addSpecialTokens: [Boolean](https://kotlinlang.org/api/core/kotlin-stdlib/kotlin/-boolean/index.html) = true): [List](https://kotlinlang.org/api/core/kotlin-stdlib/kotlin.collections/-list/index.html)&lt;[Token](../Token/index.md)&gt;

Encode [text](tokenize.md) into per-token records: each with its id, its surface text, its byte span in [text](tokenize.md), and whether the tokenizer injected it.