//clika-runtime/io.clika.modelverse/AutoTokenizer
AutoTokenizer
[common]
object AutoTokenizer
The tokenizer of the checkpoint at source (a local directory or a hub repo id; a hub source fetches the configuration and tokenizer files alone), the same tokenizer a loaded model of the checkpoint uses.
Functions
| Name | Summary |
|---|---|
| fromPretrained | [common] fun fromPretrained(source: String, options: LoadOptions = LoadOptions()): Tokenizer |