Serve it

Part 4 ended with a program that loads a checkpoint and computes once. This part puts the same model behind the serving runtime: declare its input/output contract, serve the forward pass with a lambda, and hand requests to an executor. The tutorial ends where the overview's pitch ends, a model checkpoint answering requests.

The program

C++

main.cpp

#include <cstdint>
#include <cstdio>
#include <filesystem>
#include <string>

#include <ClikaRT/clika_rt.h>

namespace rt = ClikaRT::runtime;
using ClikaRT::DataType;
using ClikaRT::NamedTensors;
using ClikaRT::Tensor;
using ClikaRT::spec::TensorSpec;
namespace io = ClikaRT::io;
namespace ops = ClikaRT::ops;

namespace {

// The layer's geometry, the checkpoint's names, and the edge names the schema
// declares and every request and response addresses. A misspelled edge name
// routes nothing, silently, so each is spelled once.
constexpr std::int64_t kFeatures = 8;
constexpr std::int64_t kOutputs = 4;
constexpr std::int64_t kBatch = 2;
constexpr const char* kWeight = "mlp.weight";
constexpr const char* kBias = "mlp.bias";
constexpr const char* kX = "x";
constexpr const char* kY = "y";

}  // namespace

int main() {
    // The checkpoint from part 4, re-created so this program stands alone.
    NamedTensors weights;
    weights.set(kWeight, Tensor::full({kOutputs, kFeatures}, 0.5, DataType::Float32));
    weights.set(kBias,   Tensor::full({kOutputs}, 0.25, DataType::Float32));
    const std::string ckpt =
        (std::filesystem::temp_directory_path() / "first_program.safetensors").string();
    io::save_safetensors(weights, ckpt);

    // Load it and declare the model's I/O contract: batches of 8 features
    // in, batches of 4 activations out. kDynamicDim leaves the batch open.
    const NamedTensors loaded = io::load_safetensors(ckpt);
    const Tensor w = loaded.get(kWeight);
    const Tensor b = loaded.get(kBias);

    rt::ModelSchema schema;
    schema.inputs  = {TensorSpec{kX, DataType::Float32, {TensorSpec::kDynamicDim, kFeatures}, false}};
    schema.outputs = {TensorSpec{kY, DataType::Float32, {TensorSpec::kDynamicDim, kOutputs}, false}};

    // Part 4's forward pass, served by a lambda. No subclass needed.
    rt::FunctionModel model{schema};
    model.on_run_once("run", [&](rt::PhaseContext& ctx) {
        const Tensor x = ctx.inputs->get(kX);
        ctx.outputs->set(kY, ops::relu(ops::linear(x, w, b)));
    });

    rt::Executor exec = rt::Executor::create(model);  // borrowed; keep `model` alive

    // One request in, one response out: the serving loop in miniature.
    rt::Request req;
    req.inputs.set(kX, Tensor::ones({kBatch, kFeatures}, DataType::Float32));
    const rt::Response resp = exec.await(exec.enqueue(std::move(req)));

    std::printf("y = %s\n", resp.outputs.get(kY).to_string().c_str());
    exec.shutdown();
    return 0;
}

Python

The Python lane serves a traced graph: crt.trace captures the forward pass as a ModelGraph, and after optimize() and finalize() the finalized graph is the servable unit. The pipeline executor over graphs is bound but not yet published, so this arm stops at the graph.

main.py

import tempfile

import clika_runtime as crt
import clika_runtime.nn.functional as F


def main() -> None:
    # The checkpoint from part 4, re-created so this program stands alone.
    ckpt = f"{tempfile.gettempdir()}/first_program.safetensors"
    crt.io.save_safetensors({
        "mlp.weight": crt.full((4, 8), 0.5),
        "mlp.bias": crt.full((4,), 0.25),
    }, ckpt)

    loaded = crt.io.load_safetensors(ckpt)
    w, b = loaded["mlp.weight"], loaded["mlp.bias"]

    # The traced callable takes and returns LISTS of tensors. crt.trace runs
    # it once over data-free stand-ins and captures the operator graph, which
    # comes back as recorded: optimize() runs the graph optimizer and
    # finalize() readies the graph to serve.
    def forward(ins: list) -> list:
        return [F.relu(F.linear(ins[0], w, b))]

    g = crt.trace(forward, example_inputs=[crt.ones(2, 8)])
    g.optimize()
    g.finalize()

    # A request in, a response out.
    (y,) = g.run([crt.ones(2, 8)])
    print(f"y = {y}")


if __name__ == "__main__":
    main()

Kotlin

The Kotlin binding does not carry the serving runtime; the C++ arm is the serving story today.

y = Tensor(shape=[2, 4], dtype=Float32, device=CPU, numel=8, data=[4.25, 4.25, 4.25, 4.25, 4.25, 4.25, ...])

The same 4.25s part 4 computed, produced this time by an executor answering a request.

Schemas, lambdas, executors

ModelSchema is the model's I/O contract: named TensorSpecs with dtype and dims, and kDynamicDim leaves a dimension open, so one served model accepts any batch size. FunctionModel serves a callable against that schema with no subclass; the lambda reads its inputs and sets its outputs through the PhaseContext. Executor::create borrows the model (keep it alive) and turns it into a queue: enqueue accepts a Request, await blocks for its Response, and shutdown drains the queue. Sessions, continuous batching and pipelines build on this same executor; the runtime example project walks each one.

One flag for the serving runtime

The serving runtime is built without RTTI, so a target that uses runtime:: adds one line to part 1's CMake (the bundle's own runtime examples set the same flag):

target_compile_options(hello PRIVATE -fno-rtti)

Next: part 6, the same tutorial program on a real phone.