Serve a model over HTTP

You have working compute and want it behind an HTTP endpoint. ClikaRT::http ships a server in the same library: declare routes with lambdas, parse and build bodies with ClikaRT::Json, and stream with server-sent events. No web framework enters the ship path.

Each program below starts a server, drives it with the built-in HTTP client in the same process, and prints the exchange, so it runs self-contained. To poke a server from outside instead, replace bind_to_any_port with listen_async("0.0.0.0", 8080) and use curl.

A served process runs compute like any other, so give it a license credential where you set the rest of its environment (CLIKA_RT_LICENSE, or the per-user file clikart-license-init writes; Get ClikaRT). Without one, a handler the runtime refuses for licensing reports the code name LICENSE_FAILED.

Start a server and add routes

Routes are declared before the server starts: get/post for fixed paths, route with a regex for path parameters (req.param(0) is the first capture). Handlers return a ServerResponse and run concurrently on the server's I/O pool, so anything they share needs a lock.

C++

routes.cpp

#include <cstdio>
#include <string>

#include "ClikaRT/clika_rt.h"

namespace http = ClikaRT::http;
using ClikaRT::json::Json;
using ClikaRT::Result;

int main() {
    http::HttpServer server = http::HttpServer::create();

    server.get("/health", [](http::ServerRequest&) -> Result<http::ServerResponse> {
        Json o = Json::object();
        o["status"] = "ok";
        o["runtime"] = ClikaRT::GetVersionInfo();
        return http::ServerResponse::json(o.dump());
    });

    server.route(http::Method::Get, R"(/models/(\w+))",
                 [](http::ServerRequest& req) -> Result<http::ServerResponse> {
                     Json o = Json::object();
                     o["model"] = req.param(0);
                     o["loaded"] = false;
                     return http::ServerResponse::json(o.dump());
                 });

    const int port = server.bind_to_any_port("127.0.0.1");
    server.wait_until_ready();
    const std::string base = "http://127.0.0.1:" + std::to_string(port);

    std::printf("GET /health        -> %s\n", http::get_text(base + "/health").c_str());
    std::printf("GET /models/smol   -> %s\n", http::get_text(base + "/models/smol").c_str());

    server.stop();
    return 0;
}

C

The C ABI carries no server, request, or SSE types; serving is the C++ tier's job (the C++ tab). The compute an endpoint wraps stays C-callable through the api table, and a served model is consumed from C with any HTTP client library.

Python

The Python server lands with the HTTP bindings of the clika_runtime.http module; the samples on this page run once the wheel carries it. Routes are decorators or calls: get / post for fixed paths, route(method, pattern, pattern=True) for a regular expression with named captures, read back with request.param(name). A handler returns an http.Response, a str (sent as text/plain) or a dict / list (sent as JSON). Handlers run on the server's own threads and hold the interpreter lock only while they run Python code.

routes.py

import http.client
import json

from clika_runtime import http as crt_http

server = crt_http.HttpServer()

@server.get("/health")
def health(request: crt_http.Request) -> dict[str, str]:
    return {"status": "ok"}                   # a dict is sent as JSON

@server.route("GET", r"/models/(?P<name>[^/]+)", pattern=True)
def model(request: crt_http.Request) -> str:
    return f"model {request.param('name')}"    # a str is sent as text/plain

@server.post("/echo")
def echo(request: crt_http.Request) -> crt_http.Response:
    return crt_http.Response.bytes(request.body, "application/octet-stream")

with server:                                   # the block ends with server.stop()
    port = server.bind_to_any_port("127.0.0.1")
    client = http.client.HTTPConnection("127.0.0.1", port)
    client.request("GET", "/health")
    print(json.loads(client.getresponse().read()))          # {'status': 'ok'}
    client.request("GET", "/models/tiny")
    print(client.getresponse().read().decode())             # model tiny
    client.request("POST", "/echo", body=b"ping")
    print(client.getresponse().read())                      # b'ping'

server.use(middleware) wraps every route with middleware(request, next); server.group(prefix) scopes routes under a path prefix; listen(host, port) blocks the calling thread until stop() and listen_async serves on the server's own thread. Every one of them releases the interpreter lock while it waits.

Kotlin

The Kotlin binding does not carry the serving runtime; the C++ arm is the serving story today. A served model is consumed from Kotlin with any JVM HTTP client, and Modelverse's serving guide shows the OpenAI-compatible route.

Go

The HTTP server and client are part of the C++ API today; the C++ tab shows the full pattern. A served model is consumed from Go with the standard library's HTTP client, and Modelverse's serving guide shows the OpenAI-compatible route.

Rust

The Rust crate wraps the C ABI, which carries no server surface; serve from the C++ tier and keep Rust on the compute side. A served model is consumed from Rust with any HTTP client crate.

GET /health        -> {"status":"ok","runtime":"0.5.1"}
GET /models/smol   -> {"model":"smol","loaded":false}

A JSON inference endpoint

The serving shape every model endpoint repeats: parse the body, validate, build a tensor from the request, compute, and put the result back into JSON. Bad input gets a clean 400 with a reason, not an exception. The compute here is one dense layer with fixed weights, standing in for a loaded model (the GGUF guide is where real weights come from). The Python tab carries the compute half for real: the same scoring model compiled once to a ModelGraph and run per request; only the HTTP transport stays C++.

C++

score_endpoint.cpp

#include <cstdint>
#include <cstdio>
#include <string>
#include <vector>

#include "ClikaRT/clika_rt.h"

namespace http = ClikaRT::http;
namespace ops = ClikaRT::ops;
using ClikaRT::DataType;
using ClikaRT::json::Json;
using ClikaRT::Result;
using ClikaRT::Tensor;

constexpr std::int64_t kFeatures = 4;

int main() {
    // The "model": y = x * W^T + b, weights fixed for a reproducible page.
    const Tensor w = Tensor::full({2, kFeatures}, 0.5, DataType::Float32);
    const Tensor b = Tensor::full({2}, 0.25, DataType::Float32);

    http::HttpServer server = http::HttpServer::create();
    server.post("/score", [&](http::ServerRequest& req) -> Result<http::ServerResponse> {
        Result<Json> body = CLIKART_TRY(Json::parse(req.body()));
        if (!body.ok())
            return http::ServerResponse::json(R"({"error":"invalid json"})", 400);
        Result<Json> feats = CLIKART_TRY(body.value().at("features"));
        if (!feats.ok() || feats.value().size() != kFeatures)
            return http::ServerResponse::json(R"({"error":"'features' must hold 4 numbers"})", 400);

        std::vector<float> x(kFeatures);
        for (std::size_t i = 0; i < kFeatures; ++i)
            x[i] = static_cast<float>(feats.value().at(i).as_double());

        const Tensor input = Tensor::from_data(x.data(), {1, kFeatures}, DataType::Float32);
        const std::vector<float> scores =
            ops::linear(input, w, b).reshape({-1}).item_as_vec<float>();

        Json out = Json::object();
        out["scores"] = Json::array();
        for (float s : scores) out["scores"].push_back(Json(static_cast<double>(s)));
        return http::ServerResponse::json(out.dump());
    });

    const int port = server.bind_to_any_port("127.0.0.1");
    server.wait_until_ready();
    const std::string base = "http://127.0.0.1:" + std::to_string(port);

    std::printf("POST /score [1,2,3,4] -> %s\n",
                http::post_text(base + "/score", R"({"features":[1, 2, 3, 4]})").c_str());
    // Client helpers raise ClikaRT::Error on any status >= 400; CLIKART_TRY
    // captures that as a Result when a failure is an expected outcome.
    Result<std::string> bad = CLIKART_TRY(http::post_text(base + "/score", R"({"features":[1]})"));
    std::printf("POST /score [1]       -> %s\n",
                bad.ok() ? bad.value().c_str() : bad.message().c_str());

    server.stop();
    return 0;
}

C

No C server surface exists, so the endpoint shell stays C++ (the C++ tab). The scoring compute itself is fully C-callable: the tensor walk-through builds exactly the tensor-in, tensor-out half a C++ endpoint wraps.

Python

score_compute.py

# The model is compiled to a ModelGraph once at startup and run per request;
# the route below is the served endpoint (the ONNX chapters of the python
# examples build bigger graphs the same way).
import json

import numpy as np

import clika_runtime as crt
from clika_runtime import http as crt_http

FEATURES = 4

# The "model": y = x * W^T + b, as a compiled graph with fixed weights.
model = crt.io.OnnxModel.create("score", opset_version=21)
model.add_input("X", crt.float32, [1, FEATURES])
w = model.add_initializer(crt.tensor(np.full((FEATURES, 2), 0.5, dtype=np.float32)))
b = model.add_initializer(crt.tensor(np.full(2, 0.25, dtype=np.float32)))
(mm,) = model.add_node("MatMul", ["X", w])
(scores,) = model.add_node("Add", [mm, b])
model.add_output(scores, crt.float32, [1, 2])
graph = model.compile()  # every node parses, weights bind, shapes resolve


def score(body: str) -> tuple[int, str]:
    """The handler shape: parse, validate, run the graph, answer JSON."""
    try:
        feats = json.loads(body).get("features")
    except ValueError:
        return 400, json.dumps({"error": "invalid json"})
    if not isinstance(feats, list) or len(feats) != FEATURES:
        return 400, json.dumps({"error": "'features' must hold 4 numbers"})

    x = crt.tensor(np.asarray([feats], dtype=np.float32))
    (y,) = graph.run({"X": x})
    return 200, json.dumps({"scores": y.numpy().reshape(-1).tolist()})


server = crt_http.HttpServer()

@server.post("/score")
def score_route(request: crt_http.Request) -> crt_http.Response:
    status, body = score(request.body.decode())
    return crt_http.Response.json(json.loads(body), status=status)

print("POST /score [1,2,3,4] ->", *score('{"features":[1, 2, 3, 4]}'))   # 200 {"scores": [5.25, 5.25]}
print("POST /score [1]       ->", *score('{"features":[1]}'))            # 400 {"error": "'features' must hold 4 numbers"}

A bad body answers a 400 with a reason, a good one a 200 with the scores; the route runs the same score the two prints call.

Kotlin

The Kotlin binding does not carry the serving runtime. The compute half is expressible (the typed tensor tier and the generated ClikaRtGen surface both reach it); the HTTP shell around it is the C++ tab's job.

Go

The HTTP transport is part of the C++ API today; the C++ tab shows it. The compute half is real in Go: open and compile a graph as in the ONNX guide, and the handler body is a graph.Run call between JSON decode and encode.

Rust

The Rust crate reaches the compute half through the f_* api-table duals; the HTTP shell around it stays on the C++ tier (the C++ tab).

POST /score [1,2,3,4] -> {"scores":[5.25,5.25]}
POST /score [1]       -> 400 Bad Request on POST /score

The handler captures the weight tensors by reference; they outlive the server. A real model swaps the ops::linear line for its forward and nothing else changes shape. On the failure, the in-process client surfaces the status as the Result's message; an external client (curl, a browser) reads the JSON error body the handler wrote.

Stream results with server-sent events

Token-by-token streaming (the transport behind LLM chat responses) is one factory away: ServerResponse::sse takes a next callback, and the server pulls it until it returns an empty optional, one SSE frame per event. The client here buffers the finite stream and prints the raw frames; a browser or an SSE-aware client consumes them incrementally.

C++

stream_tokens.cpp

#include <cstdio>
#include <memory>
#include <optional>
#include <string>

#include "ClikaRT/clika_rt.h"

namespace http = ClikaRT::http;
using ClikaRT::Result;

int main() {
    http::HttpServer server = http::HttpServer::create();

    server.get("/generate", [](http::ServerRequest&) -> Result<http::ServerResponse> {
        auto sent = std::make_shared<int>(0);  // per-connection cursor
        return http::ServerResponse::sse(
            [sent]() -> Result<std::optional<http::ServerSentEvent>> {
                static const char* kTokens[] = {"Tensors ", "stream ", "one ", "by ", "one."};
                if (*sent >= 5) return std::optional<http::ServerSentEvent>{};  // close
                http::ServerSentEvent ev;
                ev.event = "token";
                ev.data = kTokens[(*sent)++];
                return std::optional<http::ServerSentEvent>{ev};
            });
    });

    const int port = server.bind_to_any_port("127.0.0.1");
    server.wait_until_ready();

    const std::string raw =
        http::get_text("http://127.0.0.1:" + std::to_string(port) + "/generate");
    std::printf("raw SSE body:\n%s", raw.c_str());

    server.stop();
    return 0;
}

C

Server-sent events ride the C++ server, which has no C-table members. The PRODUCER half is C-callable: the streaming-decode section pushes ids through streaming_decoder_push and each non-empty piece is one frame for whatever transport carries it.

Python

A streamed body pulls its frames from a generator: crt_http.Response.sse(events) sends one event: / data: frame per item, and the server pulls the next item on its own thread as the client reads. The producer is the generation loop; the tokenizer guide's streaming-decode section turns its ids into exactly the text pieces a stream sends, one non-empty piece per token event.

sse.py

from clika_runtime import http as crt_http

server = crt_http.HttpServer()

@server.get("/events")
def events(request: crt_http.Request) -> crt_http.Response:
    def frames():
        for piece in ("Tensors ", "stream ", "one ", "by ", "one."):
            yield {"event": "token", "data": piece}   # one frame per generated piece
    return crt_http.Response.sse(frames())

The same route with a decoder behind it yields each non-empty push result of the streaming decoder as a frame's data.

Kotlin

The Kotlin binding does not carry the serving runtime. The producer half runs in Kotlin (the streaming decoder rides ClikaRtGen, see the streaming-decode section); the SSE transport is the C++ tab's job.

Go

The SSE transport is part of the C++ API today; the C++ tab shows it. The producer half is real in Go: the tokenizer guide's streaming-decode section turns a generation loop's ids into exactly the text pieces a stream like this sends.

Rust

The Rust crate reaches the streaming decoder (the producer half, see the streaming-decode section); the SSE transport stays on the C++ tier (the C++ tab).

raw SSE body:
event: token
data: Tensors 

event: token
data: stream 

event: token
data: one 

event: token
data: by 

event: token
data: one.

Each frame is an event: line, a data: line, and a blank line; a generation loop replaces the fixed token array with reads from its decoder, and the tokenizer guide's streaming-decode section is that decoder: each non-empty push result is one frame's data. The server pulls next on an I/O worker, so a slow producer stalls only its own connection.

Middleware (logging, auth), static mounts, and an image-upload endpoint that runs compute are in the bundle's http_server example, chapters 03 to 06. Wiring a served endpoint into sessions and continuous batching is the runtime example's ground. The finished version of this page's story ships in Modelverse: an OpenAI-compatible endpoint over these same server pieces, with its 01_serve example as the smallest complete server (Additional examples has it).