//clika-runtime/io.clika.runtime/Tokenizer/Companion/fromTiktoken

fromTiktoken

[common]
fun fromTiktoken(path: String): Tokenizer

Load an OpenAI .tiktoken ranks file; the special tokens of the family its table size names are added (a size outside the known families fails naming them).