//clika-runtime/io.clika.runtime/Tokenizer/tokenize

tokenize

[common]
fun tokenize(text: String, addSpecialTokens: Boolean = true): List<Token>

Encode text into per-token records: each with its id, its surface text, its byte span in text, and whether the tokenizer injected it.