Preprocess inputs with processors

Decoding a file into pixels or samples is half the input story; the other half is what the MODEL was trained to receive: resized, rescaled, normalized pixels, or a log-mel spectrogram at a fixed rate. processor::ImageProcessor and processor::AudioProcessor carry that half. Build one from explicit knobs when you know the recipe, or load it straight from the model's own preprocessor_config.json so the recipe cannot drift from the checkpoint.

Load images and audio for inference covers the decode step this page starts after; the processors consume the same tensors it produces.

Build an image pipeline from knobs

from_args covers the common recipe with no config file: resize, center-crop, rescale, normalize. The example uses a constant image so the result is checkable by hand: every input pixel 127 becomes (127/255 - 0.5) / 0.5.

image_knobs.cpp
#include <cstdio>

#include "ClikaRT/clika_rt.h"

using ClikaRT::DataType;
using ClikaRT::Tensor;
using ClikaRT::processor::ImageProcessor;
using ClikaRT::unwrap;

int main() {
    const Tensor image = Tensor::full({8, 8, 3}, 127, DataType::UInt8);

    const float mean[] = {0.5F, 0.5F, 0.5F};
    const float std_[] = {0.5F, 0.5F, 0.5F};
    ImageProcessor proc = unwrap(ImageProcessor::from_args(
        /*resize=*/{{4, 4}}, /*crop=*/{}, /*rescale=*/{},
        ClikaRT::Span<const float>(mean, 3), ClikaRT::Span<const float>(std_, 3)));

    const Tensor out = unwrap(proc.process(image));
    std::printf("features = %s\n", out.to_string().c_str());
    return 0;
}

image_knobs.py
import numpy as np
import clika_runtime as crt

image = crt.tensor(np.full((8, 8, 3), 127, dtype=np.uint8))
proc = crt.processor.ImageProcessor.from_args(
    resize=(4, 4), mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5]
)
out = proc.process(image)
print(out.dtype, out.shape)   # float32 (4, 4, 3)
print(out.numpy()[0, 0, 0])   # (127/255 - 0.5) / 0.5

features = Tensor(shape=[4, 4, 3], dtype=Float32, device=CPU, numel=48, data=[-0.003922, -0.003922, -0.003922, -0.003922, -0.003922, -0.003922, ...])

Knob semantics worth knowing: rescale defaults to 1/255; mean and std must arrive together (both present turns normalize on, both absent leaves it off); and process also takes a file path or an in-memory byte span, folding the decode step in when you have not done it yourself.

Or load the model's own recipe

A checkpoint that ships preprocessor_config.json names its exact pipeline. from_huggingface reads that file (or the model directory holding it), so preprocessing follows the checkpoint instead of a hand-copied recipe. Unrecognized keys are ignored with a logged warning naming them.

proc = crt.processor.ImageProcessor.from_huggingface("SmolVLM-Instruct")
features = proc.process("photo.jpg")   # decode + the model's own recipe

The audio front end

AudioProcessor.from_args() is the standard log-mel front end, 16 kHz and 80 mel bins by default; feed it a waveform and the sample rate it actually has.

audio_frontend.py
import numpy as np
import clika_runtime as crt

t = np.arange(16000, dtype=np.float32) / 16000.0
waveform = crt.tensor(0.5 * np.sin(2.0 * np.pi * 440.0 * t))   # 1 s of 440 Hz

audio = crt.processor.AudioProcessor.from_args()
feats = audio.process(waveform, sample_rate=16000)
print(feats.dtype, list(feats.shape))   # float32, an 80-mel-bin axis inside

The frame axis follows the hop length, so its extent depends on the clip; the 80-bin axis is the front end's signature. The same from_huggingface path exists here, reading the audio half of a checkpoint's preprocessor config.

Where this fits: decode turns files into tensors; processors turn tensors into MODEL inputs; and the tensor a processor returns feeds a forward or a compiled graph directly.

Build an image pipeline from knobs​

Or load the model's own recipe​

The audio front end​

Build an image pipeline from knobs

Or load the model's own recipe

The audio front end