---
title: "PreTokenizer"
sidebar_label: "PreTokenizer"
description: "The clika_runtime.tokenizer PreTokenizer class."
---

<!-- Generated by tools/api_reference/generate_api_docs.py. Do not edit. -->

The pre-segmentation stage of a TokenizerBuilder: cuts the text into the pieces the model tokenizes on their own (whitespace, byte-level, metaspace, regex splits). sequence() composes several in order. A builder or a sequence takes the stage; a taken stage refuses its next use with ValueError.

## `PrependScheme`

```python
PrependScheme(value, names=None, *, module=None, qualname=None, type=None, start=1)
```

Where metaspace prepends its marker; the string spellings 'always', 'first' and 'never' are accepted in its place.

## `SplitDelimiterBehavior`

```python
SplitDelimiterBehavior(value, names=None, *, module=None, qualname=None, type=None, start=1)
```

What split() does with the delimiter it matched; the string spellings 'removed', 'isolated', 'merged_with_previous', 'merged_with_next' and 'contiguous' are accepted in its place.

## `__init__`

```python
__init__(self, /, *args, **kwargs)
```

Initialize self.  See help(type(self)) for accurate signature.

## `bert`

```python
bertbert() -> clika_runtime._core.tokenizer.PreTokenizer
```

bert() -> clika_runtime._core.tokenizer.PreTokenizer

bert() -> PreTokenizer

The BERT pre-tokenizer (whitespace and punctuation).

## `byte_level`

```python
byte_levelbyte_level(add_prefix_space: bool = True, trim_offsets: bool = True, use_regex: bool = True) -> clika_runtime._core.tokenizer.PreTokenizer
```

byte_level(add_prefix_space: bool = True, trim_offsets: bool = True, use_regex: bool = True) -> clika_runtime._core.tokenizer.PreTokenizer

byte_level(add_prefix_space=True, trim_offsets=True, use_regex=True) -> PreTokenizer

The GPT-2 byte-level pre-tokenizer: add_prefix_space inserts a leading space, use_regex applies the GPT-2 token pattern.

## `digits`

```python
digitsdigits(individual_digits: bool = False) -> clika_runtime._core.tokenizer.PreTokenizer
```

digits(individual_digits: bool = False) -> clika_runtime._core.tokenizer.PreTokenizer

digits(individual_digits=False) -> PreTokenizer

Split runs of digits, each digit on its own when individual_digits.

## `metaspace`

```python
metaspacemetaspace(replacement: str = '▁', prepend_scheme: object = 'always', split: bool = True) -> clika_runtime._core.tokenizer.PreTokenizer
```

metaspace(replacement: str = '▁', prepend_scheme: object = 'always', split: bool = True) -> clika_runtime._core.tokenizer.PreTokenizer

metaspace(replacement='▁', prepend_scheme='always', split=True) -> PreTokenizer

The SentencePiece metaspace: spaces become `replacement`; prepend_scheme ('always', 'first', 'never', or a PreTokenizer.PrependScheme) says where the marker is prepended; split also breaks on the marker.

## `sequence`

```python
sequencesequence(pretokenizers: collections.abc.Sequence[clika_runtime._core.tokenizer.PreTokenizer]) -> clika_runtime._core.tokenizer.PreTokenizer
```

sequence(pretokenizers: collections.abc.Sequence[clika_runtime._core.tokenizer.PreTokenizer]) -> clika_runtime._core.tokenizer.PreTokenizer

sequence(pretokenizers) -> PreTokenizer

Apply the given pre-tokenizers in order; each is taken.

## `split`

```python
splitsplit(pattern: str, behavior: object = 'isolated', invert: bool = False, is_regex: bool = True) -> clika_runtime._core.tokenizer.PreTokenizer
```

split(pattern: str, behavior: object = 'isolated', invert: bool = False, is_regex: bool = True) -> clika_runtime._core.tokenizer.PreTokenizer

split(pattern, behavior='isolated', invert=False, is_regex=True) -> PreTokenizer

Split on `pattern` (a regular expression when is_regex, else an exact string); behavior ('removed', 'isolated', 'merged_with_previous', 'merged_with_next', 'contiguous', or a PreTokenizer.SplitDelimiterBehavior) says what happens to the delimiter; invert splits on the complement. Raises on a pattern that does not compile.

## `whitespace_split`

```python
whitespace_splitwhitespace_split() -> clika_runtime._core.tokenizer.PreTokenizer
```

whitespace_split() -> clika_runtime._core.tokenizer.PreTokenizer

whitespace_split() -> PreTokenizer

Split on whitespace only.