Skip to main content

ClikaRT::tokenizer::Normalizer

class

Header: ClikaRT/tokenizer/builder.h

The text-cleanup stage (optional): Unicode normalization, case folding, stripping, regex replacement. Composable via sequence(); each factory returns a move-only handle TokenizerBuilder::normalizer consumes.

Static member functions​

bert()​

static Normalizer bert(
    bool clean_text = true,
    bool handle_chinese_chars = true,
    bool strip_accents = false,
    bool has_strip_accents = false,
    bool lowercase = true
)

The BERT text cleaner. strip_accents unset defaults to lowercase.

Declared in ClikaRT/tokenizer/builder.h, line 143

replace()​

static Normalizer replace(
    std::string_view pattern,
    std::string_view content,
    bool is_regex = false
)

Replace every match of pattern with content. is_regex selects a regex (re2) vs an exact-string pattern; raises if a regex does not compile.

Declared in ClikaRT/tokenizer/builder.h, line 150

sequence()​

static Normalizer sequence(std::vector<Normalizer> normalizers)

Apply a list of normalizers in order (each consumed).

Declared in ClikaRT/tokenizer/builder.h, line 155

lowercase()​

static Normalizer lowercase()

Lowercase the text.

Declared in ClikaRT/tokenizer/builder.h, line 157

strip()​

static Normalizer strip(bool left = true, bool right = true)

Trim whitespace; left/right choose which ends.

Declared in ClikaRT/tokenizer/builder.h, line 159

strip_accents()​

static Normalizer strip_accents()

Remove combining accents.

Declared in ClikaRT/tokenizer/builder.h, line 161

nmt()​

static Normalizer nmt()

The NMT text cleaner (control-character scrubbing).

Declared in ClikaRT/tokenizer/builder.h, line 163

prepend()​

static Normalizer prepend(std::string_view prefix)

Prepend prefix to a non-empty string.

Declared in ClikaRT/tokenizer/builder.h, line 165

nfc()​

static Normalizer nfc()

Unicode NFC normalization.

Declared in ClikaRT/tokenizer/builder.h, line 167

nfd()​

static Normalizer nfd()

Unicode NFD normalization.

Declared in ClikaRT/tokenizer/builder.h, line 169

nfkc()​

static Normalizer nfkc()

Unicode NFKC normalization.

Declared in ClikaRT/tokenizer/builder.h, line 171

nfkd()​

static Normalizer nfkd()

Unicode NFKD normalization.

Declared in ClikaRT/tokenizer/builder.h, line 173

Member functions​

Normalizer(void)​

explicit Normalizer(void* impl) noexcept

Declared in ClikaRT/tokenizer/builder.h, line 175

Normalizer(Normalizer)​

Normalizer(Normalizer&& other) noexcept

Move-only handle; the moved-from object is empty.

Declared in ClikaRT/tokenizer/builder.h, line 177

operator=()​

Normalizer& operator=(Normalizer&& other) noexcept

Move-only handle; the moved-from object is empty.

Declared in ClikaRT/tokenizer/builder.h, line 179

~Normalizer()​

~Normalizer()

Releases the normalizer.

Declared in ClikaRT/tokenizer/builder.h, line 181