Skip to main content

Preprocess inputs with processors

Decoding a file into pixels or samples is half the input story; the other half is what the MODEL was trained to receive: resized, rescaled, normalized pixels, or a log-mel spectrogram at a fixed rate. processor::ImageProcessor and processor::AudioProcessor carry that half. Build one from explicit knobs when you know the recipe, or load it straight from the model's own preprocessor_config.json so the recipe cannot drift from the checkpoint.

Load images and audio for inference covers the decode step this page starts after; the processors consume the same tensors it produces.

Build an image pipeline from knobs​

from_args covers the common recipe with no config file: resize, center-crop, rescale, normalize. The example uses a constant image so the result is checkable by hand: every input pixel 127 becomes (127/255 - 0.5) / 0.5.

image_knobs.cpp
#include <cstdio>

#include <ClikaRT/clika_rt.h>

using ClikaRT::DataType;
using ClikaRT::Tensor;
using ClikaRT::processor::ImageProcessor;

int main() {
    const Tensor image = Tensor::full({8, 8, 3}, 127, DataType::UInt8);

    const float mean[] = {0.5F, 0.5F, 0.5F};
    const float std_[] = {0.5F, 0.5F, 0.5F};
    ImageProcessor proc = ImageProcessor::from_args(
        /*resize=*/{{4, 4}}, /*crop=*/{}, /*rescale=*/{}, mean, std_);

    const Tensor out = proc.process(image);
    std::printf("features = %s\n", out.to_string().c_str());
    return 0;
}
features = Tensor(shape=[4, 4, 3], dtype=Float32, device=CPU, numel=48, data=[-0.003922, -0.003922, -0.003922, -0.003922, -0.003922, -0.003922, ...])

Knob semantics worth knowing: rescale defaults to 1/255; mean and std must arrive together (both present turns normalize on, both absent leaves it off); and process also takes a file path or an in-memory byte span, folding the decode step in when you have not done it yourself.

Or load the model's own recipe​

A checkpoint that ships preprocessor_config.json names its exact pipeline. from_huggingface reads that file (or the model directory holding it), so preprocessing follows the checkpoint instead of a hand-copied recipe. Unrecognized keys are ignored with a logged warning naming them.

proc = crt.processor.ImageProcessor.from_huggingface("SmolVLM-Instruct")
features = proc.process("photo.jpg")   # decode + the model's own recipe

The audio front end​

AudioProcessor.from_args() is the standard log-mel front end, 16 kHz and 80 mel bins by default; feed it a waveform and the sample rate it actually has.

audio_frontend.py
import math

import clika_runtime as crt

# 1 s of a 440 Hz tone, composed on the runtime: t = n / rate, s = 0.5 sin(2 pi f t).
t = crt.arange(0.0, 16000.0, 1.0, dtype=crt.float32) / 16000.0
waveform = 0.5 * crt.sin(2.0 * math.pi * 440.0 * t)

audio = crt.processor.AudioProcessor.from_args()
feats = audio.process(waveform, sample_rate=16000)
print(feats.dtype, list(feats.shape))   # float32, an 80-mel-bin axis inside

The frame axis follows the hop length, so its extent depends on the clip; the 80-bin axis is the front end's signature. The same from_huggingface path exists here, reading the audio half of a checkpoint's preprocessor config.

Where this fits: decode turns files into tensors; processors turn tensors into MODEL inputs; and the tensor a processor returns feeds a forward or a compiled graph directly.