Skip to main content

Serve a model over HTTP

You have working compute and want it behind an HTTP endpoint. ClikaRT::http ships a server in the same library: declare routes with lambdas, parse and build bodies with ClikaRT::Json, and stream with server-sent events. No web framework enters the ship path.

Each program below starts a server, drives it with the built-in HTTP client in the same process, and prints the exchange, so it runs self-contained. To poke a server from outside instead, replace bind_to_any_port with listen_async("0.0.0.0", 8080) and use curl.

Start a server and add routes​

Routes are declared before the server starts: get/post for fixed paths, route with a regex for path parameters (req.param(0) is the first capture). Handlers return a ServerResponse and run concurrently on the server's I/O pool, so anything they share needs a lock.

routes.cpp
#include <cstdio>
#include <string>

#include "ClikaRT/clika_rt.h"

namespace http = ClikaRT::http;
using ClikaRT::json::Json;
using ClikaRT::Result;

int main() {
    http::HttpServer server = http::HttpServer::create();

    server.get("/health", [](http::ServerRequest&) -> Result<http::ServerResponse> {
        Json o = Json::object();
        o["status"] = "ok";
        o["runtime"] = ClikaRT::GetVersionInfo();
        return http::ServerResponse::json(o.dump());
    });

    server.route(http::Method::Get, R"(/models/(\w+))",
                 [](http::ServerRequest& req) -> Result<http::ServerResponse> {
                     Json o = Json::object();
                     o["model"] = req.param(0);
                     o["loaded"] = false;
                     return http::ServerResponse::json(o.dump());
                 });

    const int port = server.bind_to_any_port("127.0.0.1");
    server.wait_until_ready();
    const std::string base = "http://127.0.0.1:" + std::to_string(port);

    std::printf("GET /health        -> %s\n", http::get_text(base + "/health").c_str());
    std::printf("GET /models/smol   -> %s\n", http::get_text(base + "/models/smol").c_str());

    server.stop();
    return 0;
}
GET /health        -> {"status":"ok","runtime":"0.4.6"}
GET /models/smol   -> {"model":"smol","loaded":false}

A JSON inference endpoint​

The serving shape every model endpoint repeats: parse the body, validate, build a tensor from the request, compute, and put the result back into JSON. Bad input gets a clean 400 with a reason, not an exception. The compute here is one dense layer with fixed weights, standing in for a loaded model (the GGUF guide is where real weights come from). The Python tab carries the compute half for real: the same scoring model compiled once to a ModelGraph and run per request; only the HTTP transport stays C++.

score_endpoint.cpp
#include <cstdint>
#include <cstdio>
#include <string>
#include <vector>

#include "ClikaRT/clika_rt.h"

namespace http = ClikaRT::http;
namespace ops = ClikaRT::ops;
using ClikaRT::DataType;
using ClikaRT::json::Json;
using ClikaRT::Result;
using ClikaRT::Tensor;

constexpr std::int64_t kFeatures = 4;

int main() {
    // The "model": y = x * W^T + b, weights fixed for a reproducible page.
    const Tensor w = Tensor::full({2, kFeatures}, 0.5, DataType::Float32);
    const Tensor b = Tensor::full({2}, 0.25, DataType::Float32);

    http::HttpServer server = http::HttpServer::create();
    server.post("/score", [&](http::ServerRequest& req) -> Result<http::ServerResponse> {
        Result<Json> body = CLIKART_TRY(Json::parse(req.body()));
        if (!body.ok())
            return http::ServerResponse::json(R"({"error":"invalid json"})", 400);
        Result<Json> feats = CLIKART_TRY(body.value().at("features"));
        if (!feats.ok() || feats.value().size() != kFeatures)
            return http::ServerResponse::json(R"({"error":"'features' must hold 4 numbers"})", 400);

        std::vector<float> x(kFeatures);
        for (std::size_t i = 0; i < kFeatures; ++i)
            x[i] = static_cast<float>(feats.value().at(i).as_double());

        const Tensor input = Tensor::from_data(x.data(), {1, kFeatures}, DataType::Float32);
        const std::vector<float> scores =
            ops::linear(input, w, b).reshape({-1}).item_as_vec<float>();

        Json out = Json::object();
        out["scores"] = Json::array();
        for (float s : scores) out["scores"].push_back(Json(static_cast<double>(s)));
        return http::ServerResponse::json(out.dump());
    });

    const int port = server.bind_to_any_port("127.0.0.1");
    server.wait_until_ready();
    const std::string base = "http://127.0.0.1:" + std::to_string(port);

    std::printf("POST /score [1,2,3,4] -> %s\n",
                http::post_text(base + "/score", R"({"features":[1, 2, 3, 4]})").c_str());
    // Client helpers raise ClikaRT::Error on any status >= 400; CLIKART_TRY
    // captures that as a Result when a failure is an expected outcome.
    Result<std::string> bad = CLIKART_TRY(http::post_text(base + "/score", R"({"features":[1]})"));
    std::printf("POST /score [1]       -> %s\n",
                bad.ok() ? bad.value().c_str() : bad.message().c_str());

    server.stop();
    return 0;
}
POST /score [1,2,3,4] -> {"scores":[5.25,5.25]}
POST /score [1]       -> 400 Bad Request on POST /score

The handler captures the weight tensors by reference; they outlive the server. A real model swaps the ops::linear line for its forward and nothing else changes shape. On the failure, the in-process client surfaces the status as the Result's message; an external client (curl, a browser) reads the JSON error body the handler wrote.

Stream results with server-sent events​

Token-by-token streaming (the transport behind LLM chat responses) is one factory away: ServerResponse::sse takes a next callback, and the server pulls it until it returns an empty optional, one SSE frame per event. The client here buffers the finite stream and prints the raw frames; a browser or an SSE-aware client consumes them incrementally.

stream_tokens.cpp
#include <cstdio>
#include <memory>
#include <optional>
#include <string>

#include "ClikaRT/clika_rt.h"

namespace http = ClikaRT::http;
using ClikaRT::Result;

int main() {
    http::HttpServer server = http::HttpServer::create();

    server.get("/generate", [](http::ServerRequest&) -> Result<http::ServerResponse> {
        auto sent = std::make_shared<int>(0);  // per-connection cursor
        return http::ServerResponse::sse(
            [sent]() -> Result<std::optional<http::ServerSentEvent>> {
                static const char* kTokens[] = {"Tensors ", "stream ", "one ", "by ", "one."};
                if (*sent >= 5) return std::optional<http::ServerSentEvent>{};  // close
                http::ServerSentEvent ev;
                ev.event = "token";
                ev.data = kTokens[(*sent)++];
                return std::optional<http::ServerSentEvent>{ev};
            });
    });

    const int port = server.bind_to_any_port("127.0.0.1");
    server.wait_until_ready();

    const std::string raw =
        http::get_text("http://127.0.0.1:" + std::to_string(port) + "/generate");
    std::printf("raw SSE body:\n%s", raw.c_str());

    server.stop();
    return 0;
}
raw SSE body:
event: token
data: Tensors 

event: token
data: stream 

event: token
data: one 

event: token
data: by 

event: token
data: one.

Each frame is an event: line, a data: line, and a blank line; a generation loop replaces the fixed token array with reads from its decoder, and the tokenizer guide's streaming-decode section is that decoder: each non-empty push result is one frame's data. The server pulls next on an I/O worker, so a slow producer stalls only its own connection.

Middleware (logging, auth), static mounts, and an image-upload endpoint that runs compute are in the bundle's http_server example, chapters 03 to 06. Wiring a served endpoint into sessions and continuous batching is the runtime example's ground. The finished version of this page's story ships in Modelverse: an OpenAI-compatible endpoint over these same server pieces, with its 01_serve example as the smallest complete server (Additional examples has it).