---
title: Serve a model over HTTP
description: Put compute behind HTTP endpoints with the built-in server, answer JSON requests with tensor results, and stream tokens with server-sent events.
---

You have working compute and want it behind an HTTP endpoint. `ClikaRT::http` ships a server in the same library: declare routes with lambdas, parse and build bodies with `ClikaRT::Json`, and stream with server-sent events. No web framework enters the ship path.

Each program below starts a server, drives it with the built-in HTTP client in the same process, and prints the exchange, so it runs self-contained. To poke a server from outside instead, replace `bind_to_any_port` with `listen_async("0.0.0.0", 8080)` and use curl.

A served process runs compute like any other, so give it a license credential where you set the rest of its environment (`CLIKA_RT_LICENSE`, or the per-user file `clikart-license-init` writes; [Get ClikaRT](../getting-started/get-clikart.mdx#license-credential)). Without one, a handler the runtime refuses for licensing reports the code name `LICENSE_FAILED`.

## Start a server and add routes

Routes are declared before the server starts: `get`/`post` for fixed paths, `route` with a regex for path parameters (`req.param(0)` is the first capture). Handlers return a `ServerResponse` and run concurrently on the server's I/O pool, so anything they share needs a lock.

<LangTabs>

<LangTab value="cpp">

```cpp title="routes.cpp"
#include <cstdio>
#include <string>

#include <ClikaRT/clika_rt.h>

namespace http = ClikaRT::http;
using ClikaRT::json::Json;
using ClikaRT::Result;

int main() {
    http::HttpServer server = http::HttpServer::create();

    server.get("/health", [](http::ServerRequest&) -> Result<http::ServerResponse> {
        Json o = Json::object();
        o["status"] = "ok";
        o["runtime"] = ClikaRT::GetVersionInfo();
        return http::ServerResponse::json(o.dump());
    });

    server.route(http::Method::Get, R"(/models/(\w+))",
                 [](http::ServerRequest& req) -> Result<http::ServerResponse> {
                     Json o = Json::object();
                     o["model"] = req.param(0);
                     o["loaded"] = false;
                     return http::ServerResponse::json(o.dump());
                 });

    const int port = server.bind_to_any_port("127.0.0.1");
    server.wait_until_ready();
    const std::string base = "http://127.0.0.1:" + std::to_string(port);

    std::printf("GET /health        -> %s\n", http::get_text(base + "/health").c_str());
    std::printf("GET /models/smol   -> %s\n", http::get_text(base + "/models/smol").c_str());

    server.stop();
    return 0;
}
```

</LangTab>

<LangTab value="python">

The Python server lands with the HTTP bindings of the `clika_runtime.http` module; the samples on this page run once the wheel carries it. Routes are decorators or calls: `get` / `post` for fixed paths, `route(method, pattern, pattern=True)` for a regular expression with named captures, read back with `request.param(name)`. A handler returns an `http.Response`, a `str` (sent as `text/plain`) or a `dict` / `list` (sent as JSON). Handlers run on the server's own threads and hold the interpreter lock only while they run Python code.

```python title="routes.py"
import http.client
import json

from clika_runtime import http as crt_http

server = crt_http.HttpServer()

@server.get("/health")
def health(request: crt_http.Request) -> dict[str, str]:
    return {"status": "ok"}                   # a dict is sent as JSON

@server.route("GET", r"/models/(?P<name>[^/]+)", pattern=True)
def model(request: crt_http.Request) -> str:
    return f"model {request.param('name')}"    # a str is sent as text/plain

@server.post("/echo")
def echo(request: crt_http.Request) -> crt_http.Response:
    return crt_http.Response.bytes(request.body, "application/octet-stream")

with server:                                   # the block ends with server.stop()
    port = server.bind_to_any_port("127.0.0.1")
    client = http.client.HTTPConnection("127.0.0.1", port)
    client.request("GET", "/health")
    print(json.loads(client.getresponse().read()))          # {'status': 'ok'}
    client.request("GET", "/models/tiny")
    print(client.getresponse().read().decode())             # model tiny
    client.request("POST", "/echo", body=b"ping")
    print(client.getresponse().read())                      # b'ping'
```

`server.use(middleware)` wraps every route with `middleware(request, next)`; `server.group(prefix)` scopes routes under a path prefix; `listen(host, port)` blocks the calling thread until `stop()` and `listen_async` serves on the server's own thread. Every one of them releases the interpreter lock while it waits.

</LangTab>

<LangTab value="kotlin">

The Kotlin binding does not carry the serving runtime; the C++ arm is the serving story today. A served model is consumed from Kotlin with any JVM HTTP client, and [Modelverse's serving guide](/modelverse/how-to/serve-openai-compatible) shows the OpenAI-compatible route.

</LangTab>

</LangTabs>

```text
GET /health        -> {"status":"ok","runtime":"0.6.4"}
GET /models/smol   -> {"model":"smol","loaded":false}
```

## A JSON inference endpoint

The serving shape every model endpoint repeats: parse the body, validate, build a tensor from the request, compute, and put the result back into JSON. Bad input gets a clean 400 with a reason, not an exception. The compute here is one dense layer with fixed weights, standing in for a loaded model ([the GGUF guide](load-quantized-weights.mdx) is where real weights come from). The Python tab carries the compute half for real: the same scoring model compiled once to a `ModelGraph` and run per request; only the HTTP transport stays C++.

<LangTabs>

<LangTab value="cpp">

```cpp title="score_endpoint.cpp"
#include <cstdint>
#include <cstdio>
#include <string>
#include <vector>

#include <ClikaRT/clika_rt.h>

namespace http = ClikaRT::http;
namespace ops = ClikaRT::ops;
using ClikaRT::DataType;
using ClikaRT::json::Json;
using ClikaRT::Result;
using ClikaRT::Tensor;

constexpr std::int64_t kFeatures = 4;

int main() {
    // The "model": y = x * W^T + b, weights fixed for a reproducible page.
    const Tensor w = Tensor::full({2, kFeatures}, 0.5, DataType::Float32);
    const Tensor b = Tensor::full({2}, 0.25, DataType::Float32);

    http::HttpServer server = http::HttpServer::create();
    server.post("/score", [&](http::ServerRequest& req) -> Result<http::ServerResponse> {
        Result<Json> body = CLIKART_TRY(Json::parse(req.body()));
        if (!body.ok())
            return http::ServerResponse::json(R"({"error":"invalid json"})", 400);
        Result<Json> feats = CLIKART_TRY(body.value().at("features"));
        if (!feats.ok() || feats.value().size() != kFeatures) {
            Json err = Json::object();
            err["error"] = Json("'features' must hold " + std::to_string(kFeatures) + " numbers");
            return http::ServerResponse::json(err.dump(), 400);
        }

        std::vector<float> x(kFeatures);
        for (std::size_t i = 0; i < kFeatures; ++i)
            x[i] = static_cast<float>(feats.value().at(i).as_double());

        const Tensor input = Tensor::from_data(x.data(), {1, kFeatures}, DataType::Float32);
        const std::vector<float> scores =
            ops::linear(input, w, b).reshape({-1}).item_as_vec<float>();

        Json out = Json::object();
        out["scores"] = Json::array();
        for (float s : scores) out["scores"].push_back(Json(static_cast<double>(s)));
        return http::ServerResponse::json(out.dump());
    });

    const int port = server.bind_to_any_port("127.0.0.1");
    server.wait_until_ready();
    const std::string base = "http://127.0.0.1:" + std::to_string(port);

    std::printf("POST /score [1,2,3,4] -> %s\n",
                http::post_text(base + "/score", R"({"features":[1, 2, 3, 4]})").c_str());
    // Client helpers raise ClikaRT::Error on any status >= 400; CLIKART_TRY
    // captures that as a Result when a failure is an expected outcome.
    Result<std::string> bad = CLIKART_TRY(http::post_text(base + "/score", R"({"features":[1]})"));
    std::printf("POST /score [1]       -> %s\n",
                bad.ok() ? bad.value().c_str() : bad.message().c_str());

    server.stop();
    return 0;
}
```

</LangTab>

<LangTab value="python">

```python title="score_compute.py"
# The model is compiled to a ModelGraph once at startup and run per request;
# the route below is the served endpoint (the ONNX chapters of the python
# examples build bigger graphs the same way).
import json

import numpy as np

import clika_runtime as crt
from clika_runtime import http as crt_http

FEATURES = 4

# The "model": y = x * W^T + b, as a compiled graph with fixed weights.
model = crt.io.OnnxModel.create("score", opset_version=21)
model.add_input("X", crt.float32, [1, FEATURES])
w = model.add_initializer(crt.tensor(np.full((FEATURES, 2), 0.5, dtype=np.float32)))
b = model.add_initializer(crt.tensor(np.full(2, 0.25, dtype=np.float32)))
(mm,) = model.add_node("MatMul", ["X", w])
(scores,) = model.add_node("Add", [mm, b])
model.add_output(scores, crt.float32, [1, 2])
graph = model.compile()  # every node parses, weights bind, shapes resolve
graph.optimize()         # the graph optimizer
graph.finalize()         # only a finalized graph runs


def score(body: str) -> tuple[int, str]:
    """The handler shape: parse, validate, run the graph, answer JSON."""
    try:
        feats = json.loads(body).get("features")
    except ValueError:
        return 400, json.dumps({"error": "invalid json"})
    if not isinstance(feats, list) or len(feats) != FEATURES:
        return 400, json.dumps({"error": "'features' must hold 4 numbers"})

    x = crt.tensor(np.asarray([feats], dtype=np.float32))
    (y,) = graph.run({"X": x})
    return 200, json.dumps({"scores": y.numpy().reshape(-1).tolist()})


server = crt_http.HttpServer()

@server.post("/score")
def score_route(request: crt_http.Request) -> crt_http.Response:
    status, body = score(request.body.decode())
    return crt_http.Response.json(json.loads(body), status=status)

print("POST /score [1,2,3,4] ->", *score('{"features":[1, 2, 3, 4]}'))   # 200 {"scores": [5.25, 5.25]}
print("POST /score [1]       ->", *score('{"features":[1]}'))            # 400 {"error": "'features' must hold 4 numbers"}
```

A bad body answers a 400 with a reason, a good one a 200 with the scores; the route runs the same `score` the two prints call.

</LangTab>

<LangTab value="kotlin">

The Kotlin binding carries the serving runtime (`FunctionModel`, `Executor`, `Pipeline`) and the HTTP server (`HttpServer`); the worked program on this page is the C++ one, and [Language bindings](../bindings.md) says what Kotlin carries.

</LangTab>

</LangTabs>

```text
POST /score [1,2,3,4] -> {"scores":[5.25,5.25]}
POST /score [1]       -> 400 Bad Request on POST /score
```

The handler captures the weight tensors by reference; they outlive the server. A real model swaps the `ops::linear` line for its forward and nothing else changes shape. On the failure, the in-process client surfaces the status as the `Result`'s message; an external client (curl, a browser) reads the JSON error body the handler wrote.

## Stream results with server-sent events

Token-by-token streaming (the transport behind LLM chat responses) is one factory away: `ServerResponse::sse` takes a `next` callback, and the server pulls it until it returns an empty optional, one SSE frame per event. The client here buffers the finite stream and prints the raw frames; a browser or an SSE-aware client consumes them incrementally.

<LangTabs>

<LangTab value="cpp">

```cpp title="stream_tokens.cpp"
#include <cstdio>
#include <memory>
#include <optional>
#include <string>

#include <ClikaRT/clika_rt.h>

namespace http = ClikaRT::http;
using ClikaRT::Result;

namespace {
constexpr const char* kTokens[] = {"Tensors ", "stream ", "one ", "by ", "one."};
constexpr int kTokenCount = static_cast<int>(sizeof kTokens / sizeof kTokens[0]);
}  // namespace

int main() {
    http::HttpServer server = http::HttpServer::create();

    server.get("/generate", [](http::ServerRequest&) -> Result<http::ServerResponse> {
        auto sent = std::make_shared<int>(0);  // per-connection cursor
        return http::ServerResponse::sse(
            [sent]() -> Result<std::optional<http::ServerSentEvent>> {
                if (*sent >= kTokenCount) return std::optional<http::ServerSentEvent>{};  // close
                http::ServerSentEvent ev;
                ev.event = "token";
                ev.data = kTokens[(*sent)++];
                return std::optional<http::ServerSentEvent>{ev};
            });
    });

    const int port = server.bind_to_any_port("127.0.0.1");
    server.wait_until_ready();

    const std::string raw =
        http::get_text("http://127.0.0.1:" + std::to_string(port) + "/generate");
    std::printf("raw SSE body:\n%s", raw.c_str());

    server.stop();
    return 0;
}
```

</LangTab>

<LangTab value="python">

A streamed body pulls its frames from a generator: `crt_http.Response.sse(events)` sends one `event:` / `data:` frame per item, and the server pulls the next item on its own thread as the client reads. The producer is the generation loop; [the tokenizer guide's streaming-decode section](tokenize-and-chat-templates.mdx) turns its ids into exactly the text pieces a stream sends, one non-empty piece per `token` event.

```python title="sse.py"
from clika_runtime import http as crt_http

server = crt_http.HttpServer()

@server.get("/events")
def events(request: crt_http.Request) -> crt_http.Response:
    def frames():
        for piece in ("Tensors ", "stream ", "one ", "by ", "one."):
            yield {"event": "token", "data": piece}   # one frame per generated piece
    return crt_http.Response.sse(frames())
```

The same route with a decoder behind it yields each non-empty `push` result of the streaming decoder as a frame's `data`.

</LangTab>

<LangTab value="kotlin">

The Kotlin binding carries the serving runtime, the streaming decoder and the HTTP server; the worked program on this page is the C++ one, and [Language bindings](../bindings.md) says what Kotlin carries.

</LangTab>

</LangTabs>

```text
raw SSE body:
event: token
data: Tensors 

event: token
data: stream 

event: token
data: one 

event: token
data: by 

event: token
data: one.
```

Each frame is an `event:` line, a `data:` line, and a blank line; a generation loop replaces the fixed token array with reads from its decoder, and [the tokenizer guide's streaming-decode section](tokenize-and-chat-templates.mdx) is that decoder: each non-empty `push` result is one frame's `data`. The server pulls `next` on an I/O worker, so a slow producer stalls only its own connection.

Middleware (logging, auth), static mounts, and an image-upload endpoint that runs compute are in the bundle's `http_server` example, chapters `03` to `06`. Wiring a served endpoint into sessions and continuous batching is the `runtime` example's ground. The finished version of this page's story ships in Modelverse: [an OpenAI-compatible endpoint](/modelverse/how-to/serve-openai-compatible) over these same server pieces, with its `01_serve` example as the smallest complete server ([Additional examples](/modelverse/examples) has it).