Serve a model over the OpenAI and Anthropic APIs

Any model whose family provides serve becomes an HTTP endpoint in one command, speaking the OpenAI request and response shapes and, for chat, the Anthropic Messages API beside them. Existing OpenAI and Anthropic SDK code connects by changing its base URL; nothing else about the client changes.

Start the server

clikart-cli Qwen/Qwen2.5-0.5B-Instruct serve --host 0.0.0.0 --port 8000

[2026-10-04 07:59:45.310] [modelverse] [info] serving on http://0.0.0.0:8000 (model=Qwen2.5-0.5B-Instruct); open it in a browser for the chat page
[2026-10-04 07:59:45.310] [modelverse] [info] endpoints: POST /v1/chat/completions /v1/messages /v1/messages/count_tokens /v1/audio/transcriptions; GET /v1/models /health /props / (web UI) /dashboard
[2026-10-04 07:59:45.310] [modelverse] [info] authentication: none; every route is open

The load flags from prompt apply unchanged (--device, --max-seq, a GGUF weight selector on the source); the operational flags are the server's own:

• --host and --port: the bind address, 127.0.0.1:8000 by default. Loopback by default is deliberate; expose it with --host 0.0.0.0 when you mean to.
• --no-web-ui: disable the built-in chat page on /.
• --cors: answer cross-origin requests, for a browser front end on another origin.
• --api-key KEY: require this key on every route but /health, /v1/health, GET / and GET /dashboard, sent as Authorization: Bearer <key> (the OpenAI clients' form) or x-api-key: <key> (the Anthropic clients' form). A request without it, or with another key, answers 401 in the client's own error envelope with a WWW-Authenticate: Bearer challenge. Without the flag every route is open, so set it whenever the server listens beyond loopback; a key on the command line is readable in the process list. The two pages open without a key (a browser navigation carries no header); the chat page sends the key typed into its API key field on every call, and the dashboard reads the same stored key. The key is a best-effort gate, not a security boundary: it keeps a stray client on your network from using the model, and that is all it does. Access control for a server people you do not know can reach (identities, per-user keys and their rotation, rate limits, TLS, audit logs) belongs to an infrastructure layer in front of the serving process, a reverse proxy or an API gateway; the serving process itself stays a single-tenant engine behind it.
• --max-active and --max-queue: admission control. Requests past the queue limit are shed and surface as HTTP 503 after retries with backoff, so an overloaded server degrades loudly instead of stalling silently.
• --max-upload-mib: the largest request body accepted, 256 MiB by default. A larger upload answers 413 naming the size and the limit. Audio sizing: a 60-minute 16 kHz mono WAV is 111 MiB.
• --read-timeout: seconds a connection may stay silent while its request is read, 30 by default. A stalled read closes the connection rather than holding a slot.
• --tool-parser: how tool calls are read out of the model's text. By default the server picks, from the call formats the model's family declares, the one the checkpoint's chat template writes. The flag names another parser (serve --help lists them), or none turns extraction off. With no parser in use, a tool call stays in the reply as plain text. /props reports the parser in use as tool_parser, and tool_parser_source says where it came from: family or option.

The routes

What the server answers depends on the model's family; every serving model carries the common rows.

Route	Serves	Model families
POST /v1/chat/completions	chat, streaming (SSE) and non-streaming	text and multimodal generators
POST /v1/messages	chat in the Anthropic Messages API shape, streaming (SSE) and non-streaming	text and multimodal generators
POST /v1/messages/count_tokens	the input token count the same /v1/messages request would ingest (text only)	text and multimodal generators
POST /v1/audio/transcriptions	multipart audio file to transcript; response_format json (the default), text, verbose_json (the task, the language, the duration and the segments with their seconds and tokens), srt or vtt (one cue per segment); language the hint, temperature the decode's sampling (0, the default, is greedy; above 0 the decode samples under a fixed seed); timestamp_granularities[] takes segment (word timestamps are not produced); prompt is refused (no decode here conditions on a text prompt)	speech-to-text (Whisper)
POST /v1/audio/speech; POST, GET, DELETE /v1/voices	text in, waveform out, in a voice registered first through the voice registry (Speak text); response_format wav (the default) or pcm always, and mp3, opus, aac or flac when the host provides the ffmpeg tool (CLIKA_RT_FFMPEG names its path, else it is taken from the PATH; the tool is spawned, never linked); GET /props lists the served set as speech_formats, and the start-up log says which	text-to-speech (Chatterbox, Qwen3-TTS)
POST /v1/embeddings	texts to dense vectors; encoding_format float (the default) or base64 (each row as the base64 of its float32 little-endian bytes); dimensions shortens every row to its leading entries renormalized to unit length, refused past the model's own count	embedding models
POST /v1/similarity	texts to a cosine matrix	embedding models
POST /v1/translations	texts plus a language pair	translation models
POST /v1/images/generations, POST /v1/videos/generations	prompt to a generated image or clip; response_format b64_json (the default) or url: the file is kept in memory under an unguessable id for an hour (at most 64 files, 256 MiB in all; the oldest leave first) and served at GET /v1/files/<id> with no key, the id being the credential as a signed URL's is	diffusion models
POST /v1/depth	multipart image to depth artifacts	depth estimators
POST /v1/segmentations	multipart image to a class-mask document	segmentation models
POST /v1/detections	multipart image to box/score/label rows (an open-vocabulary detector also takes a prompt)	detectors
GET /v1/models, GET /props	what is loaded and how it is configured; the model listing answers the OpenAI shape (object, created, owned_by, license), or the Anthropic one (type, display_name, created_at, has_more) to a request carrying anthropic-version or x-api-key	all
GET /health, GET /v1/health	{"status":"ok"}, the machine probe	all
GET /	the built-in web chat page	all, unless --no-web-ui
GET /dashboard	the live request dashboard	all
any other path	404 in the client's own envelope: the OpenAI one (code unknown_url, the path named) or, under /v1/messages, the Messages one (not_found_error)	all

Call it with the OpenAI API

Non-streaming, from anything that can POST JSON:

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "One-line haiku about rain."}]}'

Streaming adds "stream": true and reads server-sent events, one data: line per delta, data: [DONE] last, the shape OpenAI clients already parse. Which means the SDKs work as-is:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="unused")
for chunk in client.chat.completions.create(
    model="qwen",
    messages=[{"role": "user", "content": "One-line haiku about rain."}],
    stream=True,
):
    print(chunk.choices[0].delta.content or "", end="", flush=True)

A multimodal model takes OpenAI-shaped image content parts in the same route; a Whisper model is called with a multipart file upload instead (Transcribe audio shows both of its forms).

Call it with the Anthropic API

POST /v1/messages takes the Messages API request body on the same server. max_tokens is required, as it is in the Anthropic API. The anthropic-version header is optional; 2023-06-01 is the version served, and any other value is refused. A request may name any model, and the reply names the one the server loaded.

curl -s http://127.0.0.1:8000/v1/messages \
  -H "Content-Type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -d '{"model": "qwen", "max_tokens": 256,
       "messages": [{"role": "user", "content": "One-line haiku about rain."}]}'

The Anthropic SDK takes the server's address as its base URL, without /v1. Any API key works, because the server checks none:

from anthropic import Anthropic

client = Anthropic(base_url="http://127.0.0.1:8000", api_key="unused")
with client.messages.stream(
    model="qwen",
    max_tokens=256,
    messages=[{"role": "user", "content": "One-line haiku about rain."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

A streamed reply follows the Messages API event order. It opens with message_start and a ping. Each content block then streams as content_block_start, its content_block_delta events and content_block_stop. The reply closes with message_delta, which carries the stop reason and the output usage, and then message_stop. There is no [DONE] line. A ping also goes out whenever the stream has been silent for 5 seconds.

max_tokens is a cap. When the context window runs out first, the reply stops there with stop_reason model_context_window_exceeded. A prompt that fills the window on its own is refused with a message that starts prompt is too long:. The other stop reasons are end_turn, max_tokens, stop_sequence (with the matched stop_sequence) and tool_use. In usage, input_tokens counts the prompt tokens the prefix cache did not serve, cache_read_input_tokens counts the ones it did, and output_tokens counts the reply.

Errors come back in the Anthropic error envelope, with request_id in the body, and every answer carries a request-id header. A refusal is a 400 invalid_request_error whose message starts with the path of the field it names. The other statuses:

• 404 not_found_error: a path under /v1/messages that no route serves, such as the batches API.
• 413 request_too_large: a body past --max-upload-mib.
• 529 overloaded_error: the server is at capacity, for example when admission control sheds the request. The answer carries retry-after and x-should-retry: true.
• 503 api_error: the device faulted. Restart the server; the answer carries x-should-retry: false.
• 500 api_error: the generation failed.

The anthropic-version header is read when present and taken as the served version when absent (a client that omits it is answered, not refused).

What the Messages API route serves, accepts and refuses

Request	What the server does
text; base64 images (JPEG, PNG, GIF, WebP) to a multimodal model	served
system as a string or as text blocks	served
stop_sequences (up to 4, each up to 128 bytes), temperature, top_p, top_k	served
custom tools, tool_choice auto or none, tool_use and tool_result blocks	served; calls come back as tool_use blocks when a parser reads the model's call format (the family's own by default, or the one --tool-parser names)
thinking enabled (a budget_tokens of at least 1024, below max_tokens), adaptive or disabled; output_config.effort at a level the model declares	served by a model that reasons
a final assistant turn (a response prefill: text only, with no trailing whitespace)	served; the reply carries only the new text
cache_control	accepted, no effect; the server's prefix cache needs no marker
metadata, service_tier, output_config.task_budget	accepted, no effect
context_management	accepted, not applied; the history renders in full
a tool's strict and defer_loading; the tool-search server tools	accepted; arguments are not schema-constrained, and every tool is offered up front
an effort level the model does not declare	accepted; the model's default applies
fields the server does not know, in a request that sends anthropic-beta	ignored
tool_choice any or tool (a forced tool call)	refused
other server tools, container, output_config.format (structured output)	refused
document, search_result and other block types the server does not serve	refused
an image file source or an image URL (the server fetches no remote payload)	refused
max_tokens 0	refused
fields the server does not know, in a request without anthropic-beta	refused: <field>: Extra inputs are not permitted
a response prefill with thinking enabled or adaptive, or to a model that always reasons; thinking disabled on a model that always reasons and declares no effort levels	refused
an image in a count_tokens request	refused; it counts text only

The server logs context_management, strict, defer_loading, the tool-search tools and an effort level the model does not declare once each, at Info, naming the field.

Use it from Claude Code

Claude Code speaks the Messages API, so it runs against this server once its environment variables point it there. Start the server with a context window that holds Claude Code's prompt, and put your own device in --device:

clikart-cli Qwen/Qwen3-30B-A3B-Instruct-2507 serve --device vulkan:0 --max-seq 65536

Then start Claude Code with these variables:

unset ANTHROPIC_API_KEY
export ANTHROPIC_BASE_URL=http://127.0.0.1:8000
export ANTHROPIC_AUTH_TOKEN=unused
export ANTHROPIC_MODEL=Qwen3-30B-A3B-Instruct-2507
export ANTHROPIC_DEFAULT_HAIKU_MODEL=Qwen3-30B-A3B-Instruct-2507
export CLAUDE_CODE_MAX_CONTEXT_TOKENS=65536
export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1
export API_TIMEOUT_MS=600000
claude

• ANTHROPIC_BASE_URL is the server's address, without /v1.
• ANTHROPIC_AUTH_TOKEN is the server's --api-key value, or any value when the server runs without one. Claude Code sends it as an Authorization: Bearer header, and it ranks above ANTHROPIC_API_KEY in Claude Code's credential order. Unsetting ANTHROPIC_API_KEY keeps an Anthropic key out of a session that talks to another server.
• ANTHROPIC_MODEL names the main model, and ANTHROPIC_DEFAULT_HAIKU_MODEL names the one Claude Code uses for background tasks. Behind a custom base URL, Claude Code passes any model name through unchecked, and this server answers every request with the model it loaded, so both name the served model, the id /props reports as model_id.
• CLAUDE_CODE_MAX_CONTEXT_TOKENS is the window Claude Code assumes for the model. For a model name it does not recognize, Claude Code can assume a window that differs from the one this server serves, so set it to the window /props reports as n_ctx. Claude Code then compacts the conversation before it reaches the server's limit (model configuration).
• CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 turns off Claude Code's other traffic, such as auto-updates, telemetry and error reporting, for a network that reaches only this server.
• API_TIMEOUT_MS is how long Claude Code waits for a reply, in milliseconds. 600000 (10 minutes) is its default; raise it when the first prefill of Claude Code's prompt takes longer than that on your device.

To check the connection, run /status in Claude Code. Its Anthropic base URL line shows this server's address, and its Auth token line names ANTHROPIC_AUTH_TOKEN (verify the connection).

Claude Code sends its whole system prompt and every tool definition with each request, about 16,000 tokens before the conversation starts, so the served window must hold them. The server caps the window at 4096 tokens by default, and with that cap Claude Code refuses the request itself with "Prompt is too long". --max-seq sets the window, and /props reports it as n_ctx. The tools reach the model through its chat template, and its calls come back to Claude Code as tool_use blocks when a parser reads the model's call format: the family's own by default, or the one --tool-parser names (above). /props shows which parser is in use.

With ANTHROPIC_BASE_URL pointing at a host other than api.anthropic.com, Claude Code turns off features such as Remote Control and server-managed settings (feature availability). Anthropic's gateway documentation states that Anthropic "doesn't support routing Claude Code to non-Claude models through any gateway" (LLM gateways).

Operate it

GET /health is the readiness probe for a load balancer. GET /props reports the loaded model and its effective options, the remote twin of -v's effective-options report; its device field names the device the model actually landed on, which is what you check after --device auto or a fallback. Its routes list gives every API route the server answers, each with its method and its path, plus api (openai or anthropic) on a route that speaks one of those two APIs. The dashboard on /dashboard shows the same routes and the live requests; /api/requests is its JSON feed. Everything the server logs goes to stderr like the rest of the clikart-cli executable, so systemd or a container runtime captures it without configuration.

For the same server inside your own process (your engines, your routes, the built-in UI swapped for your page), the 01_serve program in Additional examples is the smallest complete consumer of the library's ServeApi.