Skip to main content

Your first pipeline

Part 4 puts the pieces together into the shape every real ClikaRT program has: load weights and input from disk, place them on a device, compute, read results back. It is self-contained. Stage 1 writes the files it needs, standing in for a real export.

The program​

main.cpp
#include <cstdint>
#include <cstdio>
#include <filesystem>
#include <string>
#include <vector>

#include <ClikaRT/clika_rt.h>

using ClikaRT::DataType;
using ClikaRT::Device;
using ClikaRT::NamedTensors;
using ClikaRT::Tensor;
namespace io = ClikaRT::io;
namespace ops = ClikaRT::ops;

namespace {

// The layer's geometry and the checkpoint's names, each spelled once.
constexpr std::int64_t kFeatures = 8;   // columns of x, columns of W
constexpr std::int64_t kOutputs = 4;    // rows of W, the activations out
constexpr std::int64_t kBatch = 2;
constexpr const char* kWeight = "mlp.weight";
constexpr const char* kBias = "mlp.bias";

Device pick_device() {                    // from part 3
    namespace device = ClikaRT::device;
    if (device::is_cuda_available())   return Device::cuda();
    if (device::is_vulkan_available()) return Device::vulkan();
    if (device::is_metal_available())  return Device::metal();
    return Device::cpu();
}

}  // namespace

int main() {
    const std::filesystem::path dir = std::filesystem::temp_directory_path();
    const std::string ckpt  = (dir / "first_program.safetensors").string();
    const std::string input = (dir / "first_program_input.npy").string();

    // ── Stage 1: write the artifacts (a stand-in for a real export) ──
    // A checkpoint is a NamedTensors: a name -> tensor map.
    NamedTensors weights;
    weights.set(kWeight, Tensor::full({kOutputs, kFeatures}, 0.5, DataType::Float32));
    weights.set(kBias,   Tensor::full({kOutputs}, 0.25, DataType::Float32));
    io::save_safetensors(weights, ckpt);

    io::save_npy(Tensor::ones({kBatch, kFeatures}, DataType::Float32), input);  // two rows of ones

    // ── Stage 2: load, compute, read back ──
    // Loaders take the target device: the tensors land there directly.
    const Device dev = pick_device();
    const NamedTensors loaded = io::load_safetensors(ckpt, dev);
    const Tensor x = io::load_npy(input, dev);

    // One MLP layer: y = relu(x * W^T + b), [2,8] x [4,8]^T -> [2,4].
    const Tensor y = ops::relu(ops::linear(x, loaded.get(kWeight), loaded.get(kBias)));

    // Per-row mean, then back to the host (the reads from part 3).
    const std::vector<float> pooled = ops::mean(y, {1}).item_as_vec<float>();

    std::printf("y = %s\n", y.to_string().c_str());
    std::printf("row means = [%.2f, %.2f] (expected 8*1*0.5 + 0.25 = 4.25)\n",
                pooled[0], pooled[1]);
    return 0;
}

On a machine with an NVIDIA GPU (the device line follows what pick_device() found):

y = Tensor(shape=[2, 4], dtype=Float32, device=CUDA:0, numel=8, data=[4.25, 4.25, 4.25, 4.25, 4.25, 4.25, ...])
row means = [4.25, 4.25] (expected 8*1*0.5 + 0.25 = 4.25)

Every element of y is 8 * 1 * 0.5 + 0.25 = 4.25, so both row means print 4.25, on whatever device the machine offered.

Model checkpoints are NamedTensors​

A model's weights travel as a NamedTensors, a name -> tensor map with set / get / size / for_each. io::save_safetensors / io::load_safetensors round-trip it; io::load_gguf returns the same map plus the file's metadata; io::load_npy reads a single array. The loaders take the target device, so weights stream to where they will be used with no separate .to() step. That is the placement lesson from part 3, folded into I/O.

The pipeline shape​

Stage 2 is the skeleton to keep: discover -> load onto the device -> compute -> read back. Scaling it up changes the sizes, not the shape (more names in the checkpoint, a deeper stack of ops:: calls between load and read). The pieces this series did not need live in the examples. The runtime project wraps this exact shape in sessions, continuous batching and pipelines, and io's later chapters cover GGUF and images.

You have a complete, device-portable ClikaRT program. Next: part 5, the same model behind the serving runtime.