AutoTokenizer

Load a checkpoint's tokenizer by name, without its weights.

from_pretrained

from_pretrained(source: 'str', *, revision: 'str' = 'main', cache_dir: 'str | None' = None, token: 'str | None' = None, offline: 'bool' = False) -> 'Any'

The tokenizer of the checkpoint at source (a local directory or a hub repo id): the same tokenizer object a loaded model exposes as model.tokenizer. A hub source fetches the tokenizer files only.