Skip to main content

PreTokenizer

The pre-segmentation stage of a TokenizerBuilder: cuts the text into the pieces the model tokenizes on their own (whitespace, byte-level, metaspace, regex splits). sequence() composes several in order. A builder or a sequence takes the stage; a taken stage refuses its next use with ValueError.

PrependScheme​

PrependScheme(value, names=None, *, module=None, qualname=None, type=None, start=1)

Where metaspace prepends its marker; the string spellings 'always', 'first' and 'never' are accepted in its place.

SplitDelimiterBehavior​

SplitDelimiterBehavior(value, names=None, *, module=None, qualname=None, type=None, start=1)

What split() does with the delimiter it matched; the string spellings 'removed', 'isolated', 'merged_with_previous', 'merged_with_next' and 'contiguous' are accepted in its place.

__init__​

__init__(self, /, *args, **kwargs)

Initialize self. See help(type(self)) for accurate signature.

bert​

bertbert() -> clika_runtime._core.tokenizer.PreTokenizer

bert() -> clika_runtime._core.tokenizer.PreTokenizer

bert() -> PreTokenizer

The BERT pre-tokenizer (whitespace and punctuation).

byte_level​

byte_levelbyte_level(add_prefix_space: bool = True, trim_offsets: bool = True, use_regex: bool = True) -> clika_runtime._core.tokenizer.PreTokenizer

byte_level(add_prefix_space: bool = True, trim_offsets: bool = True, use_regex: bool = True) -> clika_runtime._core.tokenizer.PreTokenizer

byte_level(add_prefix_space=True, trim_offsets=True, use_regex=True) -> PreTokenizer

The GPT-2 byte-level pre-tokenizer: add_prefix_space inserts a leading space, use_regex applies the GPT-2 token pattern.

digits​

digitsdigits(individual_digits: bool = False) -> clika_runtime._core.tokenizer.PreTokenizer

digits(individual_digits: bool = False) -> clika_runtime._core.tokenizer.PreTokenizer

digits(individual_digits=False) -> PreTokenizer

Split runs of digits, each digit on its own when individual_digits.

metaspace​

metaspacemetaspace(replacement: str = '▁', prepend_scheme: object = 'always', split: bool = True) -> clika_runtime._core.tokenizer.PreTokenizer

metaspace(replacement: str = '▁', prepend_scheme: object = 'always', split: bool = True) -> clika_runtime._core.tokenizer.PreTokenizer

metaspace(replacement='▁', prepend_scheme='always', split=True) -> PreTokenizer

The SentencePiece metaspace: spaces become replacement; prepend_scheme ('always', 'first', 'never', or a PreTokenizer.PrependScheme) says where the marker is prepended; split also breaks on the marker.

sequence​

sequencesequence(pretokenizers: collections.abc.Sequence[clika_runtime._core.tokenizer.PreTokenizer]) -> clika_runtime._core.tokenizer.PreTokenizer

sequence(pretokenizers: collections.abc.Sequence[clika_runtime._core.tokenizer.PreTokenizer]) -> clika_runtime._core.tokenizer.PreTokenizer

sequence(pretokenizers) -> PreTokenizer

Apply the given pre-tokenizers in order; each is taken.

split​

splitsplit(pattern: str, behavior: object = 'isolated', invert: bool = False, is_regex: bool = True) -> clika_runtime._core.tokenizer.PreTokenizer

split(pattern: str, behavior: object = 'isolated', invert: bool = False, is_regex: bool = True) -> clika_runtime._core.tokenizer.PreTokenizer

split(pattern, behavior='isolated', invert=False, is_regex=True) -> PreTokenizer

Split on pattern (a regular expression when is_regex, else an exact string); behavior ('removed', 'isolated', 'merged_with_previous', 'merged_with_next', 'contiguous', or a PreTokenizer.SplitDelimiterBehavior) says what happens to the delimiter; invert splits on the complement. Raises on a pattern that does not compile.

whitespace_split​

whitespace_splitwhitespace_split() -> clika_runtime._core.tokenizer.PreTokenizer

whitespace_split() -> clika_runtime._core.tokenizer.PreTokenizer

whitespace_split() -> PreTokenizer

Split on whitespace only.