//clika-runtime/io.clika.runtime/Tokenizer/Companion/fromGpt2

fromGpt2

[common]
fun fromGpt2(directory: String): Tokenizer

Load a GPT-2 style byte-level BPE tokenizer from a directory holding vocab.json and merges.txt.