Serve an OpenAI-compatible endpoint

Any model whose family provides serve becomes an HTTP endpoint in one command, speaking the OpenAI request and response shapes. Existing OpenAI SDK code connects by changing its base URL; nothing else about the client changes.

Start the server

clika-modelverse meta-llama/Llama-3.2-1B-Instruct serve --host 0.0.0.0 --port 8000

[17:19:07.555] [modelverse] [info] serving on http://0.0.0.0:8000 (model=Llama-3.2-1B-Instruct); open it in a browser for the chat page
[17:19:07.555] [modelverse] [info] endpoints: POST /v1/chat/completions /v1/audio/transcriptions; GET /v1/models /health /props / (web UI) /dashboard

The load flags from prompt apply unchanged (--device, --max-seq, a GGUF weight selector on the source); the operational flags are the server's own:

• --host and --port: the bind address, 127.0.0.1:8000 by default. Loopback by default is deliberate; expose it with --host 0.0.0.0 when you mean to.
• --no-web-ui: disable the built-in chat page on /.
• --cors: answer cross-origin requests, for a browser front end on another origin.
• --max-active and --max-queue: admission control. Requests past the queue limit are shed and surface as HTTP 503 after retries with backoff, so an overloaded server degrades loudly instead of stalling silently.

The routes

What the server answers depends on the model's family; every serving model carries the common rows.

Route	Serves	Model families
POST /v1/chat/completions	chat, streaming (SSE) and non-streaming	text and multimodal generators
POST /v1/audio/transcriptions	multipart audio file to transcript	speech-to-text (Whisper)
POST /v1/audio/speech; POST, GET, DELETE /v1/voices	text in, waveform out, in a voice registered first through the voice registry (Speak text)	text-to-speech (Chatterbox, Qwen3-TTS)
POST /v1/embeddings	texts to dense vectors	embedding models
POST /v1/similarity	texts to a cosine matrix	embedding models
POST /v1/translations	texts plus a language pair	translation models
POST /v1/images/generations	prompt to generated image	diffusion models
POST /v1/depth	multipart image to depth artifacts	depth estimators
POST /v1/segmentations	multipart image to a class-mask document	segmentation models
POST /v1/detections	multipart image to box/score/label rows (an open-vocabulary detector also takes a prompt)	detectors
GET /v1/models, GET /props	what is loaded and how it is configured	all
GET /health, GET /v1/health	{"status":"ok"}, the machine probe	all
GET /	the built-in web chat page	all, unless --no-web-ui
GET /dashboard	the live request dashboard	all

Call it

Non-streaming, from anything that can POST JSON:

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "One-line haiku about rain."}]}'

Streaming adds "stream": true and reads server-sent events, one data: line per delta, data: [DONE] last, the shape OpenAI clients already parse. Which means the SDKs work as-is:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="unused")
for chunk in client.chat.completions.create(
    model="llama",
    messages=[{"role": "user", "content": "One-line haiku about rain."}],
    stream=True,
):
    print(chunk.choices[0].delta.content or "", end="", flush=True)

A multimodal model takes OpenAI-shaped image content parts in the same route; a Whisper model is called with a multipart file upload instead (Transcribe audio shows both of its forms).

Operate it

GET /health is the readiness probe for a load balancer. GET /props reports the loaded model and its effective options, the remote twin of -v's effective-options report. The dashboard on /dashboard shows live requests; /api/requests is its JSON feed. Everything the server logs goes to stderr like the rest of the clika-modelverse executable, so systemd or a container runtime captures it without configuration.

For the same server inside your own process (your engines, your routes, the built-in UI swapped for your page), the 01_serve program in Additional examples is the smallest complete consumer of the library's OpenAiServer.