--- title: Serve a model over the OpenAI and Anthropic APIs description: Host any runnable model behind the built-in HTTP server, stream completions, and point existing OpenAI and Anthropic clients at it. --- Any model whose family provides `serve` becomes an HTTP endpoint in one command, speaking the OpenAI request and response shapes and, for chat, the Anthropic Messages API beside them. Existing OpenAI and Anthropic SDK code connects by changing its base URL; nothing else about the client changes. ## Start the server ```bash clikart-cli Qwen/Qwen2.5-0.5B-Instruct serve --host 0.0.0.0 --port 8000 ``` ```text [2026-10-04 07:59:45.310] [modelverse] [info] serving on http://0.0.0.0:8000 (model=Qwen2.5-0.5B-Instruct); open it in a browser for the chat page [2026-10-04 07:59:45.310] [modelverse] [info] endpoints: POST /v1/chat/completions /v1/messages /v1/messages/count_tokens /v1/audio/transcriptions; GET /v1/models /health /props / (web UI) /dashboard [2026-10-04 07:59:45.310] [modelverse] [info] authentication: none; every route is open ``` The load flags from `prompt` apply unchanged (`--device`, `--max-seq`, a GGUF weight selector on the source); the operational flags are the server's own: - `--host` and `--port`: the bind address, `127.0.0.1:8000` by default. Loopback by default is deliberate; expose it with `--host 0.0.0.0` when you mean to. - `--no-web-ui`: disable the built-in chat page on `/`. - `--cors`: answer cross-origin requests, for a browser front end on another origin. - `--api-key KEY`: require this key on every route but `/health`, `/v1/health`, `GET /` and `GET /dashboard`, sent as `Authorization: Bearer ` (the OpenAI clients' form) or `x-api-key: ` (the Anthropic clients' form). A request without it, or with another key, answers 401 in the client's own error envelope with a `WWW-Authenticate: Bearer` challenge. Without the flag every route is open, so set it whenever the server listens beyond loopback; a key on the command line is readable in the process list. The two pages open without a key (a browser navigation carries no header); the chat page sends the key typed into its API key field on every call, and the dashboard reads the same stored key. **The key is a best-effort gate, not a security boundary:** it keeps a stray client on your network from using the model, and that is all it does. Access control for a server people you do not know can reach (identities, per-user keys and their rotation, rate limits, TLS, audit logs) belongs to an infrastructure layer in front of the serving process, a reverse proxy or an API gateway; the serving process itself stays a single-tenant engine behind it. - `--max-active` and `--max-queue`: admission control. Requests past the queue limit are shed and surface as HTTP 503 after retries with backoff, so an overloaded server degrades loudly instead of stalling silently. - `--max-upload-mib`: the largest request body accepted, 256 MiB by default. A larger upload answers 413 naming the size and the limit. Audio sizing: a 60-minute 16 kHz mono WAV is 111 MiB. - `--read-timeout`: seconds a connection may stay silent while its request is read, 30 by default. A stalled read closes the connection rather than holding a slot. - `--tool-parser`: how tool calls are read out of the model's text. By default the server picks, from the call formats the model's family declares, the one the checkpoint's chat template writes. The flag names another parser (`serve --help` lists them), or `none` turns extraction off. With no parser in use, a tool call stays in the reply as plain text. `/props` reports the parser in use as `tool_parser`, and `tool_parser_source` says where it came from: `family` or `option`. ## The routes What the server answers depends on the model's family; every serving model carries the common rows. | Route | Serves | Model families | | --- | --- | --- | | `POST /v1/chat/completions` | chat, streaming (SSE) and non-streaming | text and multimodal generators | | `POST /v1/messages` | chat in the Anthropic Messages API shape, streaming (SSE) and non-streaming | text and multimodal generators | | `POST /v1/messages/count_tokens` | the input token count the same `/v1/messages` request would ingest (text only) | text and multimodal generators | | `POST /v1/audio/transcriptions` | multipart audio file to transcript; `response_format` `json` (the default), `text`, `verbose_json` (the task, the language, the duration and the segments with their seconds and tokens), `srt` or `vtt` (one cue per segment); `language` the hint, `temperature` the decode's sampling (0, the default, is greedy; above 0 the decode samples under a fixed seed); `timestamp_granularities[]` takes `segment` (word timestamps are not produced); `prompt` is refused (no decode here conditions on a text prompt) | speech-to-text (Whisper) | | `POST /v1/audio/speech`; `POST`, `GET`, `DELETE /v1/voices` | text in, waveform out, in a voice registered first through the voice registry ([Speak text](speak-text.md)); `response_format` `wav` (the default) or `pcm` always, and `mp3`, `opus`, `aac` or `flac` when the host provides the `ffmpeg` tool (`CLIKA_RT_FFMPEG` names its path, else it is taken from the PATH; the tool is spawned, never linked); `GET /props` lists the served set as `speech_formats`, and the start-up log says which | text-to-speech (Chatterbox, Qwen3-TTS) | | `POST /v1/embeddings` | texts to dense vectors; `encoding_format` `float` (the default) or `base64` (each row as the base64 of its float32 little-endian bytes); `dimensions` shortens every row to its leading entries renormalized to unit length, refused past the model's own count | embedding models | | `POST /v1/similarity` | texts to a cosine matrix | embedding models | | `POST /v1/translations` | texts plus a language pair | translation models | | `POST /v1/images/generations`, `POST /v1/videos/generations` | prompt to a generated image or clip; `response_format` `b64_json` (the default) or `url`: the file is kept in memory under an unguessable id for an hour (at most 64 files, 256 MiB in all; the oldest leave first) and served at `GET /v1/files/` with no key, the id being the credential as a signed URL's is | diffusion models | | `POST /v1/depth` | multipart image to depth artifacts | depth estimators | | `POST /v1/segmentations` | multipart image to a class-mask document | segmentation models | | `POST /v1/detections` | multipart image to box/score/label rows (an open-vocabulary detector also takes a `prompt`) | detectors | | `GET /v1/models`, `GET /props` | what is loaded and how it is configured; the model listing answers the OpenAI shape (`object`, `created`, `owned_by`, `license`), or the Anthropic one (`type`, `display_name`, `created_at`, `has_more`) to a request carrying `anthropic-version` or `x-api-key` | all | | `GET /health`, `GET /v1/health` | `{"status":"ok"}`, the machine probe | all | | `GET /` | the built-in web chat page | all, unless `--no-web-ui` | | `GET /dashboard` | the live request dashboard | all | | any other path | 404 in the client's own envelope: the OpenAI one (`code` `unknown_url`, the path named) or, under `/v1/messages`, the Messages one (`not_found_error`) | all | ## Call it with the OpenAI API Non-streaming, from anything that can POST JSON: ```bash curl -s http://127.0.0.1:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"messages": [{"role": "user", "content": "One-line haiku about rain."}]}' ``` Streaming adds `"stream": true` and reads server-sent events, one `data:` line per delta, `data: [DONE]` last, the shape OpenAI clients already parse. Which means the SDKs work as-is: ```python from openai import OpenAI client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="unused") for chunk in client.chat.completions.create( model="qwen", messages=[{"role": "user", "content": "One-line haiku about rain."}], stream=True, ): print(chunk.choices[0].delta.content or "", end="", flush=True) ``` A multimodal model takes OpenAI-shaped image content parts in the same route; a Whisper model is called with a multipart file upload instead ([Transcribe audio](transcribe-audio.md) shows both of its forms). ## Call it with the Anthropic API `POST /v1/messages` takes the Messages API request body on the same server. `max_tokens` is required, as it is in the Anthropic API. The `anthropic-version` header is optional; `2023-06-01` is the version served, and any other value is refused. A request may name any model, and the reply names the one the server loaded. ```bash curl -s http://127.0.0.1:8000/v1/messages \ -H "Content-Type: application/json" \ -H "anthropic-version: 2023-06-01" \ -d '{"model": "qwen", "max_tokens": 256, "messages": [{"role": "user", "content": "One-line haiku about rain."}]}' ``` The Anthropic SDK takes the server's address as its base URL, without `/v1`. Any API key works, because the server checks none: ```python from anthropic import Anthropic client = Anthropic(base_url="http://127.0.0.1:8000", api_key="unused") with client.messages.stream( model="qwen", max_tokens=256, messages=[{"role": "user", "content": "One-line haiku about rain."}], ) as stream: for text in stream.text_stream: print(text, end="", flush=True) ``` A streamed reply follows the Messages API event order. It opens with `message_start` and a `ping`. Each content block then streams as `content_block_start`, its `content_block_delta` events and `content_block_stop`. The reply closes with `message_delta`, which carries the stop reason and the output usage, and then `message_stop`. There is no `[DONE]` line. A `ping` also goes out whenever the stream has been silent for 5 seconds. `max_tokens` is a cap. When the context window runs out first, the reply stops there with `stop_reason` `model_context_window_exceeded`. A prompt that fills the window on its own is refused with a message that starts `prompt is too long:`. The other stop reasons are `end_turn`, `max_tokens`, `stop_sequence` (with the matched `stop_sequence`) and `tool_use`. In `usage`, `input_tokens` counts the prompt tokens the prefix cache did not serve, `cache_read_input_tokens` counts the ones it did, and `output_tokens` counts the reply. Errors come back in the Anthropic error envelope, with `request_id` in the body, and every answer carries a `request-id` header. A refusal is a 400 `invalid_request_error` whose message starts with the path of the field it names. The other statuses: - 404 `not_found_error`: a path under `/v1/messages` that no route serves, such as the batches API. - 413 `request_too_large`: a body past `--max-upload-mib`. - 529 `overloaded_error`: the server is at capacity, for example when admission control sheds the request. The answer carries `retry-after` and `x-should-retry: true`. - 503 `api_error`: the device faulted. Restart the server; the answer carries `x-should-retry: false`. - 500 `api_error`: the generation failed. The `anthropic-version` header is read when present and taken as the served version when absent (a client that omits it is answered, not refused). ### What the Messages API route serves, accepts and refuses | Request | What the server does | | --- | --- | | text; base64 images (JPEG, PNG, GIF, WebP) to a multimodal model | served | | `system` as a string or as text blocks | served | | `stop_sequences` (up to 4, each up to 128 bytes), `temperature`, `top_p`, `top_k` | served | | custom tools, `tool_choice` `auto` or `none`, `tool_use` and `tool_result` blocks | served; calls come back as `tool_use` blocks when a parser reads the model's call format (the family's own by default, or the one `--tool-parser` names) | | `thinking` `enabled` (a `budget_tokens` of at least 1024, below `max_tokens`), `adaptive` or `disabled`; `output_config.effort` at a level the model declares | served by a model that reasons | | a final assistant turn (a response prefill: text only, with no trailing whitespace) | served; the reply carries only the new text | | `cache_control` | accepted, no effect; the server's prefix cache needs no marker | | `metadata`, `service_tier`, `output_config.task_budget` | accepted, no effect | | `context_management` | accepted, not applied; the history renders in full | | a tool's `strict` and `defer_loading`; the tool-search server tools | accepted; arguments are not schema-constrained, and every tool is offered up front | | an effort level the model does not declare | accepted; the model's default applies | | fields the server does not know, in a request that sends `anthropic-beta` | ignored | | `tool_choice` `any` or `tool` (a forced tool call) | refused | | other server tools, `container`, `output_config.format` (structured output) | refused | | `document`, `search_result` and other block types the server does not serve | refused | | an image `file` source or an image URL (the server fetches no remote payload) | refused | | `max_tokens` 0 | refused | | fields the server does not know, in a request without `anthropic-beta` | refused: `: Extra inputs are not permitted` | | a response prefill with `thinking` `enabled` or `adaptive`, or to a model that always reasons; `thinking` `disabled` on a model that always reasons and declares no effort levels | refused | | an image in a `count_tokens` request | refused; it counts text only | The server logs `context_management`, `strict`, `defer_loading`, the tool-search tools and an effort level the model does not declare once each, at Info, naming the field. ## Use it from Claude Code Claude Code speaks the Messages API, so it runs against this server once its environment variables point it there. Start the server with a context window that holds Claude Code's prompt, and put your own device in `--device`: ```bash clikart-cli Qwen/Qwen3-30B-A3B-Instruct-2507 serve --device vulkan:0 --max-seq 65536 ``` Then start Claude Code with these variables: ```bash unset ANTHROPIC_API_KEY export ANTHROPIC_BASE_URL=http://127.0.0.1:8000 export ANTHROPIC_AUTH_TOKEN=unused export ANTHROPIC_MODEL=Qwen3-30B-A3B-Instruct-2507 export ANTHROPIC_DEFAULT_HAIKU_MODEL=Qwen3-30B-A3B-Instruct-2507 export CLAUDE_CODE_MAX_CONTEXT_TOKENS=65536 export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 export API_TIMEOUT_MS=600000 claude ``` - `ANTHROPIC_BASE_URL` is the server's address, without `/v1`. - `ANTHROPIC_AUTH_TOKEN` is the server's `--api-key` value, or any value when the server runs without one. Claude Code sends it as an `Authorization: Bearer` header, and it ranks above `ANTHROPIC_API_KEY` in Claude Code's [credential order](https://code.claude.com/docs/en/authentication). Unsetting `ANTHROPIC_API_KEY` keeps an Anthropic key out of a session that talks to another server. - `ANTHROPIC_MODEL` names the main model, and `ANTHROPIC_DEFAULT_HAIKU_MODEL` names the one Claude Code uses for background tasks. Behind a custom base URL, Claude Code passes any model name through unchecked, and this server answers every request with the model it loaded, so both name the served model, the id `/props` reports as `model_id`. - `CLAUDE_CODE_MAX_CONTEXT_TOKENS` is the window Claude Code assumes for the model. For a model name it does not recognize, Claude Code can assume a window that differs from the one this server serves, so set it to the window `/props` reports as `n_ctx`. Claude Code then compacts the conversation before it reaches the server's limit ([model configuration](https://code.claude.com/docs/en/model-config)). - `CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1` turns off Claude Code's other traffic, such as auto-updates, telemetry and error reporting, for a network that reaches only this server. - `API_TIMEOUT_MS` is how long Claude Code waits for a reply, in milliseconds. 600000 (10 minutes) is its default; raise it when the first prefill of Claude Code's prompt takes longer than that on your device. To check the connection, run `/status` in Claude Code. Its `Anthropic base URL` line shows this server's address, and its `Auth token` line names `ANTHROPIC_AUTH_TOKEN` ([verify the connection](https://code.claude.com/docs/en/llm-gateway-connect#verify-the-connection)). Claude Code sends its whole system prompt and every tool definition with each request, about 16,000 tokens before the conversation starts, so the served window must hold them. The server caps the window at 4096 tokens by default, and with that cap Claude Code refuses the request itself with "Prompt is too long". `--max-seq` sets the window, and `/props` reports it as `n_ctx`. The tools reach the model through its chat template, and its calls come back to Claude Code as `tool_use` blocks when a parser reads the model's call format: the family's own by default, or the one `--tool-parser` names (above). `/props` shows which parser is in use. With `ANTHROPIC_BASE_URL` pointing at a host other than `api.anthropic.com`, Claude Code turns off features such as Remote Control and server-managed settings ([feature availability](https://code.claude.com/docs/en/feature-availability)). Anthropic's gateway documentation states that Anthropic "doesn't support routing Claude Code to non-Claude models through any gateway" ([LLM gateways](https://code.claude.com/docs/en/llm-gateway)). ## Operate it `GET /health` is the readiness probe for a load balancer. `GET /props` reports the loaded model and its effective options, the remote twin of `-v`'s effective-options report; its `device` field names the device the model actually landed on, which is what you check after `--device auto` or a fallback. Its `routes` list gives every API route the server answers, each with its method and its path, plus `api` (`openai` or `anthropic`) on a route that speaks one of those two APIs. The dashboard on `/dashboard` shows the same routes and the live requests; `/api/requests` is its JSON feed. Everything the server logs goes to stderr like the rest of the `clikart-cli` executable, so systemd or a container runtime captures it without configuration. For the same server inside your own process (your engines, your routes, the built-in UI swapped for your page), the `01_serve` program in [Additional examples](../examples.md) is the smallest complete consumer of the library's `ServeApi`.