Skip to main content

Serve and chat

The model generates on demand; this part keeps it loaded. serve hosts the model behind an OpenAI-compatible server with a built-in chat page. One command, no configuration files:

clika-modelverse meta-llama/Llama-3.2-1B-Instruct serve
[2026-09-03 22:42:03.787] [modelverse] [info] serving on http://127.0.0.1:8000 (model=Llama-3.2-1B-Instruct); open it in a browser for the chat page
[2026-09-03 22:42:03.787] [modelverse] [info] endpoints: POST /v1/chat/completions /v1/audio/transcriptions; GET /v1/models /health /props / (web UI) /dashboard

The defaults bind 127.0.0.1:8000; --host 0.0.0.0 opens it to the network and --port moves it. Three things are now running:

  • The API. POST /v1/chat/completions, streaming (server-sent events) and non-streaming, in the OpenAI request and response shape.
  • A web chat page. http://127.0.0.1:8000/ serves a built-in chat UI from inside the binary (--no-web-ui disables it).
  • A health probe. GET /health answers {"status":"ok"}, for load balancers and scripts.

Chat in the browser​

The chat page is the conversation surface: a scrolling transcript with an input box, the assistant reply streaming token by token as the pipeline decodes it. The conversation carries its history, so follow-up questions see earlier turns. The same conversation is available to any OpenAI client through the API, where the system message is the first messages entry and the sampling knobs (temperature, top_p) ride each request.

One templated turn from the terminal stays prompt, part 2's command. Releases before 0.4.3 also carried a terminal chat verb; it is gone, and a chat on the command line is refused like any unknown verb. prompt, serve and bench are the text-generation surface.

Call it over HTTP​

From a second terminal:

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "messages": [{"role": "user", "content": "Say hello in French."}],
        "max_tokens": 32
      }'
{"id":"chatcmpl-1788475323-0","object":"chat.completion","created":1788475323,"model":"Llama-3.2-1B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"Bonjour ! Comment allez-vous ?"},"finish_reason":"stop"}],"usage":{"prompt_tokens":40,"completion_tokens":8,"total_tokens":48}}

The id, created and usage values are your run's own. Add "stream": true and the response arrives as server-sent events, one delta per chunk, the way OpenAI clients expect. An existing client needs one change:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="unused")
reply = client.chat.completions.create(
    model="llama",
    messages=[{"role": "user", "content": "Say hello in French."}],
)
print(reply.choices[0].message.content)

The endpoint surface is bigger than chat: a Whisper model's serve answers POST /v1/audio/transcriptions, an embedding model's answers POST /v1/embeddings, and so on per family. Serve an OpenAI-compatible endpoint has the full route table and the operational flags.

Next: part 4, the same model inside your own program.