Skip to main content

Load images and audio for inference

Vision and audio models consume tensors, and ClikaRT::io gets you there from the files themselves: load_image decodes the common encoded formats (PNG, JPEG and friends) into an [H, W, C] tensor, and load_audio decodes a stream into a Float32 waveform, resampling and remixing to the layout you name. Both also take an in-memory byte span, so the same calls serve an upload handler as well as a file path. The write direction ships too: encode_audio and save_audio turn a waveform back into a WAV (Write audio back).

The runs below use a small RGB PNG (input.png, 8x6, a color gradient) and a half-second stereo WAV (clip.wav, 8 kHz, a 440 Hz tone on the left channel); any image or audio file of yours works the same. The C samples abbreviate the api-table bootstrap that tutorial part 1 shows in full.

Decode an image and make it model-ready​

peek_image reads only the header, the cheap way to validate and route before decoding. load_image decodes to [H, W, C] with 8-bit pixels; requested_channels forces a channel count (3 collapses an alpha channel or expands grayscale, so one call normalizes mixed inputs). The rest of the preprocessing every vision model wants (float, scale, CHW, batch dimension) is three ops.

image_to_input.cpp
#include <cstdio>

#include "ClikaRT/clika_rt.h"

using ClikaRT::DataType;
using ClikaRT::Tensor;
namespace io = ClikaRT::io;
namespace ops = ClikaRT::ops;

int main() {
    const io::ImageInfo info = io::peek_image("input.png");
    std::printf("header: %lldx%lld, %lld channel(s)\n", (long long)info.width,
                (long long)info.height, (long long)info.channels);

    const Tensor img = io::load_image("input.png", /*requested_channels=*/3);
    std::printf("decoded: %s\n", img.to_string().c_str());

    // Model-ready: float in [0,1], channels first, leading batch dim.
    const Tensor x = ops::unsqueeze(
        ops::permute(img.to(DataType::Float32) * (1.0 / 255.0), {2, 0, 1}), 0);
    std::printf("input:   %s\n", x.to_string().c_str());
    std::printf("mean pixel value: %.4f\n", ops::mean(x).item<float>());
    return 0;
}
header: 8x6, 3 channel(s)
decoded: Tensor(shape=[6, 8, 3], dtype=UInt8, device=CPU, numel=144, data=[0, 0, 128, 32, 0, 128, ...])
input:   Tensor(shape=[1, 3, 6, 8], dtype=Float32, device=CPU, numel=144, data=[0, 0.1255, 0.251, 0.3765, 0.502, 0.6275, ...])
mean pixel value: 0.4444

The decoded tensor is ordinary compute-engine currency from that point on: .to(device) moves it, and a model's forward takes it as-is. For bytes already in memory (an HTTP upload, an asset in an archive), pass a Span<const std::uint8_t> instead of the path; the http_server example's 06_image_compute chapter runs exactly that flow behind an upload endpoint.

Decode audio at the model's sample rate​

Audio checkpoints are trained at a fixed sample rate and channel count, and load_audio meets them at the file: target_sample_rate resamples and target_channels remixes during the decode, so the tensor that comes out is already the model's input layout. The result is an AudioData: samples (Float32, [frames] mono or [frames, channels]) plus the effective AudioInfo.

audio_to_input.cpp
#include <cstdio>

#include "ClikaRT/clika_rt.h"

using ClikaRT::Tensor;
namespace io = ClikaRT::io;
namespace ops = ClikaRT::ops;

int main() {
    // Whatever the file's native layout, ask for 16 kHz mono.
    const io::AudioData audio = io::load_audio("clip.wav", /*target_sample_rate=*/16000,
                                               /*target_channels=*/1);
    std::printf("decoded: %d Hz, %d channel(s), %lld frames (%.2f s)\n",
                audio.info.sample_rate, audio.info.channels,
                (long long)audio.info.frames,
                (double)audio.info.frames / audio.info.sample_rate);
    std::printf("samples: %s\n", audio.samples.to_string().c_str());

    // Loudness check: RMS over the waveform.
    const Tensor rms = ops::sqrt(ops::mean(audio.samples * audio.samples));
    std::printf("rms = %.4f\n", rms.item<float>());
    return 0;
}
decoded: 16000 Hz, 1 channel(s), 8000 frames (0.50 s)
samples: Tensor(shape=[8000], dtype=Float32, device=CPU, numel=8000, data=[0, 0, 0, 0.002914, 0.01748, 0.04778, ...])
rms = 0.1285

Passing 0 for either target keeps the file's native rate or channel count, and peek_audio's AudioInfo answers "what would this decode produce" without decoding. From here the waveform feeds a feature front end (ops:: has the FFT-adjacent reductions and windowed views a spectrogram needs) or goes straight into a model that consumes raw samples.

Write audio back​

The write direction is two calls: encode_audio(samples, sample_rate) returns the PCM16 WAV bytes of a waveform, and save_audio(samples, sample_rate, path) writes the same bytes to a file. A Float16, BFloat16, Float32 or Float64 waveform converts to 16-bit PCM; Int16 is written as is. [frames] is mono and [frames, channels] is interleaved multi-channel, the same layouts the decoder produces, so a model's synthesized speech goes to disk with one call and an HTTP handler returns the encoded bytes without touching a file.

audio_write.cpp
#include <cstdio>
#include <vector>

#include <ClikaRT/clika_rt.h>

using ClikaRT::DataType;
using ClikaRT::Tensor;
namespace io = ClikaRT::io;
namespace ops = ClikaRT::ops;

int main() {
    // Half a second of a 440 Hz tone at 16 kHz: t = frame / rate,
    // s = 0.5 * sin(2 pi f t), a Float32 waveform in [-1, 1].
    const int rate = 16000;
    const Tensor t = ops::arange(0.0, 8000.0, 1.0, DataType::Float32) * (1.0 / rate);
    const Tensor tone = ops::sin(t * (2.0 * 3.14159265358979 * 440.0)) * 0.5;

    // encode_audio returns the PCM16 WAV bytes; save_audio writes the same
    // bytes to a file.
    const std::vector<unsigned char> wav_bytes = io::encode_audio(tone, rate);
    std::printf("encoded: %zu bytes of RIFF/WAVE\n", wav_bytes.size());

    io::save_audio(tone, rate, "tone.wav");

    // Round trip: decode what was written and check the layout survived.
    const io::AudioData back = io::load_audio("tone.wav", rate, /*target_channels=*/1);
    std::printf("reloaded: %d Hz, %d channel(s), %lld frames\n",
                back.info.sample_rate, back.info.channels,
                static_cast<long long>(back.info.frames));
    const Tensor rms = ops::sqrt(ops::mean(back.samples * back.samples));
    std::printf("rms = %.4f\n", rms.item<float>());
    return 0;
}
encoded: 16044 bytes of RIFF/WAVE
reloaded: 16000 Hz, 1 channel(s), 8000 frames
rms = 0.3535

The byte count is the 8000 frames as 16-bit PCM plus the 44-byte RIFF header, and the round-trip RMS is the tone's own (0.5 amplitude over root two).

Decoding gives you pixels and samples; the model-specific half (resize, normalize, log-mel) is the processors guide. Weights travel the same road in the GGUF guide; the bundle's io example walks NumPy and safetensors round-trips in its earlier chapters.