---
title: "ClikaRT::tokenizer::Normalizer"
sidebar_label: "Normalizer"
description: "The text-cleanup stage (optional): Unicode normalization, case folding, stripping, regex replacement."
---

<!-- Generated by tools/doxygen2mdx from Doxygen XML. Do not edit. -->

<span class="api-kind api-kind--class">class</span>

**Header:** `ClikaRT/tokenizer/builder.h`

The text-cleanup stage (optional): Unicode normalization, case folding, stripping, regex replacement. Composable via [`sequence()`](./Normalizer.md#sequence); each factory returns a move-only handle [`TokenizerBuilder::normalizer`](./TokenizerBuilder.md#normalizer) consumes.

## Static member functions

### bert() {#bert}

<div class="api-signature">

`static` [`Normalizer`](./Normalizer.md) **`bert`**(<br />
&nbsp;&nbsp;&nbsp;&nbsp;`bool` `clean_text` `=` `true`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`bool` `handle_chinese_chars` `=` `true`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`bool` `strip_accents` `=` `false`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`bool` `has_strip_accents` `=` `false`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`bool` `lowercase` `=` `true`<br />
)

</div>

The BERT text cleaner. [`strip_accents`](./Normalizer.md#strip_accents) unset defaults to [`lowercase`](./Normalizer.md#lowercase).

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 143

</div>

### replace() {#replace}

<div class="api-signature">

`static` [`Normalizer`](./Normalizer.md) **`replace`**(<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::string_view` `pattern`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`std::string_view` `content`,<br />
&nbsp;&nbsp;&nbsp;&nbsp;`bool` `is_regex` `=` `false`<br />
)

</div>

Replace every match of `pattern` with `content`. `is_regex` selects a regex (re2) vs an exact-string pattern; raises if a regex does not compile.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 150

</div>

### sequence() {#sequence}

<div class="api-signature">

`static` [`Normalizer`](./Normalizer.md) **`sequence`**(`std::vector<`[`Normalizer`](./Normalizer.md)`>` `normalizers`)

</div>

Apply a list of normalizers in order (each consumed).

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 155

</div>

### lowercase() {#lowercase}

<div class="api-signature">

`static` [`Normalizer`](./Normalizer.md) **`lowercase`**()

</div>

Lowercase the text.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 157

</div>

### strip() {#strip}

<div class="api-signature">

`static` [`Normalizer`](./Normalizer.md) **`strip`**(`bool` `left` `=` `true`, `bool` `right` `=` `true`)

</div>

Trim whitespace; `left`/`right` choose which ends.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 159

</div>

### strip\_accents() {#strip_accents}

<div class="api-signature">

`static` [`Normalizer`](./Normalizer.md) **`strip_accents`**()

</div>

Remove combining accents.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 161

</div>

### nmt() {#nmt}

<div class="api-signature">

`static` [`Normalizer`](./Normalizer.md) **`nmt`**()

</div>

The NMT text cleaner (control-character scrubbing).

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 163

</div>

### prepend() {#prepend}

<div class="api-signature">

`static` [`Normalizer`](./Normalizer.md) **`prepend`**(`std::string_view` `prefix`)

</div>

Prepend `prefix` to a non-empty string.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 165

</div>

### nfc() {#nfc}

<div class="api-signature">

`static` [`Normalizer`](./Normalizer.md) **`nfc`**()

</div>

Unicode NFC normalization.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 167

</div>

### nfd() {#nfd}

<div class="api-signature">

`static` [`Normalizer`](./Normalizer.md) **`nfd`**()

</div>

Unicode NFD normalization.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 169

</div>

### nfkc() {#nfkc}

<div class="api-signature">

`static` [`Normalizer`](./Normalizer.md) **`nfkc`**()

</div>

Unicode NFKC normalization.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 171

</div>

### nfkd() {#nfkd}

<div class="api-signature">

`static` [`Normalizer`](./Normalizer.md) **`nfkd`**()

</div>

Unicode NFKD normalization.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 173

</div>

## Member functions

### Normalizer(void) {#Normalizer}

<div class="api-signature">

`explicit` **`Normalizer`**(`void*` `impl`) `noexcept`

</div>

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 175

</div>

### Normalizer(Normalizer) {#Normalizer-2}

<div class="api-signature">

**`Normalizer`**(`Normalizer&&` `other`) `noexcept`

</div>

Move-only handle; the moved-from object is empty.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 177

</div>

### operator=() {#operator-assign}

<div class="api-signature">

[`Normalizer`](./Normalizer.md)`&` **`operator=`**([`Normalizer`](./Normalizer.md)`&&` `other`) `noexcept`

</div>

Move-only handle; the moved-from object is empty.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 179

</div>

### \~Normalizer() {#Normalizer-3}

<div class="api-signature">

**`~Normalizer`**()

</div>

Releases the normalizer.

<div class="api-declared">

Declared in `ClikaRT/tokenizer/builder.h`, line 181

</div>