Serve it

Part 4 ended with a program that loads a checkpoint and computes once. This part puts the same model behind the serving runtime: declare its input/output contract, serve the forward pass with a lambda, and hand requests to an executor. The tutorial ends where the overview's pitch ends, a model checkpoint answering requests.

The program

C++

main.cpp

#include <cstdint>
#include <cstdio>
#include <filesystem>
#include <string>

#include <ClikaRT/clika_rt.h>

namespace rt = ClikaRT::runtime;
using ClikaRT::DataType;
using ClikaRT::NamedTensors;
using ClikaRT::Tensor;
using ClikaRT::spec::TensorSpec;
namespace io = ClikaRT::io;
namespace ops = ClikaRT::ops;

namespace {

// The layer's geometry, the checkpoint's names, and the edge names the schema
// declares and every request and response addresses. A misspelled edge name
// routes nothing, silently, so each is spelled once.
constexpr std::int64_t kFeatures = 8;
constexpr std::int64_t kOutputs = 4;
constexpr std::int64_t kBatch = 2;
constexpr const char* kWeight = "mlp.weight";
constexpr const char* kBias = "mlp.bias";
constexpr const char* kX = "x";
constexpr const char* kY = "y";

}  // namespace

int main() {
    // The checkpoint from part 4, re-created so this program stands alone.
    NamedTensors weights;
    weights.set(kWeight, Tensor::full({kOutputs, kFeatures}, 0.5, DataType::Float32));
    weights.set(kBias,   Tensor::full({kOutputs}, 0.25, DataType::Float32));
    const std::string ckpt =
        (std::filesystem::temp_directory_path() / "first_program.safetensors").string();
    io::save_safetensors(weights, ckpt);

    // Load it and declare the model's I/O contract: batches of 8 features
    // in, batches of 4 activations out. kDynamicDim leaves the batch open.
    const NamedTensors loaded = io::load_safetensors(ckpt);
    const Tensor w = loaded.get(kWeight);
    const Tensor b = loaded.get(kBias);

    rt::ModelSchema schema;
    schema.inputs  = {TensorSpec{kX, DataType::Float32, {TensorSpec::kDynamicDim, kFeatures}, false}};
    schema.outputs = {TensorSpec{kY, DataType::Float32, {TensorSpec::kDynamicDim, kOutputs}, false}};

    // Part 4's forward pass, served by a lambda. No subclass needed.
    rt::FunctionModel model{schema};
    model.on_run_once("run", [&](rt::PhaseContext& ctx) {
        const Tensor x = ctx.inputs->get(kX);
        ctx.outputs->set(kY, ops::relu(ops::linear(x, w, b)));
    });

    rt::Executor exec = rt::Executor::create(model);  // borrowed; keep `model` alive

    // One request in, one response out: the serving loop in miniature.
    rt::Request req;
    req.inputs.set(kX, Tensor::ones({kBatch, kFeatures}, DataType::Float32));
    const rt::Response resp = exec.await(exec.enqueue(std::move(req)));

    std::printf("y = %s\n", resp.outputs.get(kY).to_string().c_str());
    exec.shutdown();
    return 0;
}

C

The C core surface runs compiled graphs (the model_graph_run_vector family); the FunctionModel and Executor family is C++-only today.

Python

The Python lane serves a traced graph: crt.trace captures the forward pass as a ModelGraph, the runnable, servable unit. The pipeline executor over graphs is bound but not yet published, so this arm stops at the graph.

main.py

import tempfile

import clika_runtime as crt
import clika_runtime.nn.functional as F


def main() -> None:
    # The checkpoint from part 4, re-created so this program stands alone.
    ckpt = f"{tempfile.gettempdir()}/first_program.safetensors"
    crt.io.save_safetensors({
        "mlp.weight": crt.full((4, 8), 0.5),
        "mlp.bias": crt.full((4,), 0.25),
    }, ckpt)

    loaded = crt.io.load_safetensors(ckpt)
    w, b = loaded["mlp.weight"], loaded["mlp.bias"]

    # The traced callable takes and returns LISTS of tensors. crt.trace runs
    # it once over data-free stand-ins and captures the operator graph.
    def forward(ins: list) -> list:
        return [F.relu(F.linear(ins[0], w, b))]

    g = crt.trace(forward, example_inputs=[crt.ones(2, 8)])

    # A request in, a response out.
    (y,) = g.run([crt.ones(2, 8)])
    print(f"y = {y}")


if __name__ == "__main__":
    main()

Kotlin

The Kotlin binding does not carry the serving runtime; the C++ arm is the serving story today.

Go

The Go module rides the C core surface, which runs compiled graphs; the serving runtime stays on the C++ side.

Rust

The Rust crate does not carry the serving runtime; the C++ arm is the serving story today.

y = Tensor(shape=[2, 4], dtype=Float32, device=CPU, numel=8, data=[4.25, 4.25, 4.25, 4.25, 4.25, 4.25, ...])

The same 4.25s part 4 computed, produced this time by an executor answering a request.

Schemas, lambdas, executors

ModelSchema is the model's I/O contract: named TensorSpecs with dtype and dims, and kDynamicDim leaves a dimension open, so one served model accepts any batch size. FunctionModel serves a callable against that schema with no subclass; the lambda reads its inputs and sets its outputs through the PhaseContext. Executor::create borrows the model (keep it alive) and turns it into a queue: enqueue accepts a Request, await blocks for its Response, and shutdown drains the queue. Sessions, continuous batching and pipelines build on this same executor; the runtime example project walks each one.

One flag for the serving runtime

The serving runtime is built without RTTI, so a target that uses runtime:: adds one line to part 1's CMake (the bundle's own runtime examples set the same flag):

target_compile_options(hello PRIVATE -fno-rtti)

Next: part 6, the same tutorial program on a real phone.