Server

The ServeAPI, an OpenAI-compatible HTTP server over one text-generation model, one text-to-image model or one text-to-video model.

Over a :class:`GenerativeModel` (or a chat engine) the server answers
``POST /v1/chat/completions`` and the Anthropic Messages API
(``POST /v1/messages``, streaming and not, and
``POST /v1/messages/count_tokens``) and renders the chat page; over an
:class:`ImageGenerationModel` (or an image-generation engine) it answers
``POST /v1/images/generations`` (``b64_json``) and renders the image
page; over a :class:`VideoGenerationModel` (or a video-generation
engine) it answers ``POST /v1/videos/generations`` (the clip as base64
MP4 with its soundtrack as base64 WAV; optional first / last keyframes)
and renders the video page, one generation at a time; ``codec`` names
the clip's video codec (empty = the runtime's first available encoder).
``max_active``, ``max_queued`` and ``default_budget`` are the chat
engine's knobs; an image model generates ``max_active`` images
at once, one denoising run per request.

``start`` binds and serves on the library's thread and returns once the
server answers; ``stop`` is bounded by one decode step (every in-flight
request is canceled at its next step); the context-manager form starts
on enter and stops on exit. ``base_url`` is what an OpenAI client points
at (``/v1``-prefixed routes), and what an Anthropic SDK client, or Claude
Code through ``ANTHROPIC_BASE_URL``, points at.

``api_key``, when non-empty, is the key every request must carry, as
``Authorization: Bearer <key>`` or ``x-api-key: <key>``, on every route
but the health probes, the pages (a browser navigation carries no header;
their scripts send the key the person typed) and a CORS preflight; a
request without it answers 401 with a ``WWW-Authenticate: Bearer``
challenge in its own envelope. It is a best-effort gate (real access
control belongs to a gateway in front of the server) and is never logged.

base_url (property)

device_faulted (property)

True once the served model's device faulted for good; the server refuses every request from then on.

engine (property)

The chat engine the server front-ends.

host (property)

http (property)

The server's own HTTP server, for routes and middleware beside the model routes: server.http.get("/ext/hello", handler), server.http.use(middleware), server.http.set_exception_handler. Register before start(), since the routes are fixed once the server listens, on a path the server does not own (a reserved one raises ValueError). The handle is released at stop(): a call on it after that raises RuntimeError. Listening, binding and stopping stay with this Server.

port (property)

The bound port (for port=0, the free port chosen at start).

running (property)

__init__

__init__(self, model_or_engine: 'GenerativeModel | ImageGenerationModel | VideoGenerationModel | Any', *, host: 'str' = '127.0.0.1', port: 'int' = 8000, model_id: 'str' = '', max_active: 'int' = 1, max_queued: 'int' = 256, default_budget: 'int' = 256, enable_web_ui: 'bool' = True, enable_cors: 'bool' = False, log_request_timing: 'bool' = False, max_upload_bytes: 'int' = 0, read_timeout_seconds: 'int' = 0, codec: 'str' = '', api_key: 'str' = '') -> 'None'

Initialize self. See help(type(self)) for accurate signature.

close

close(self) -> 'None'

Stop (if serving) and release the server and, when this server built it, its engine.

serve_forever

serve_forever(self) -> 'None'

Block until :meth:stop is called from another thread or the process gets Ctrl-C; stops the server on the way out.

start

start(self) -> "'Server'"

Bind and serve; returns once the server answers. A port in use raises RuntimeError.

stop

stop(self) -> 'None'

Stop serving: in-flight requests end at their next decode step and every server thread is joined. Idempotent. A borrowed http handle is released first: a call on it afterwards raises RuntimeError.

wait_until_ready

wait_until_ready(self, timeout: 'float' = 30.0) -> 'None'

Block until GET /health answers, or raise TimeoutError after timeout seconds.