//clika-runtime/io.clika.runtime/Tokenizer/Companion/fromTekken

fromTekken

[common]
fun fromTekken(path: String): Tokenizer

Load a Mistral tekken.json tokenizer, which carries its own special tokens.