Skip to main content

Model deployment

A model deployment runs one of your registered models on one or more of your devices and puts an HTTP endpoint in front of it. The model loads once and stays loaded, so requests are answered without paying the load cost each time.

The web application calls this surface Deployments, and the Deploy button on a benchmark result leads to it. In the API the object is a serving.

Why it exists​

A benchmark tells you how a model performs on a device. A deployment is how you then use it, with an application, a notebook or a colleague sending requests and the model answering from the hardware you chose. It is also the shortest path from "this model won the benchmark" to "this model is answering", because the platform builds the whole service for you: it stages the runner, starts the process, watches its health, restarts it when it dies, and gives you an address to call.

What you choose​

A deployment takes a model, the devices to run it on, and how much of each device to use.

ChoiceWhat it does
ModelAny model registered in the project.
DevicesOne or more. Each device runs its own copy of the model and answers on its own endpoint.
Computecpu or gpu. The web application infers it from what the device reported (a device with a GPU gets gpu) rather than asking.
Context capThe maximum sequence length the server will accept. It defaults to a conservative 4096 so that a large-context checkpoint does not exhaust memory as it loads, and it is settable through the API.

Deploying from the web application means create and start in one action. Through the API the two are separate: a serving is created pending, and a start call brings it up.

The proxy​

The model server on a device binds the device's loopback interface, so nothing outside the device can reach it directly. Every call goes through the platform instead, at a per-device address:

/api/v1/servings/{serving_id}/devices/{device_id}/proxy/{path}

Everything after proxy/ is passed through to the model server verbatim, the query string travels with it, and responses stream rather than being buffered, so token-by-token output arrives as it is produced.

Three things are worth knowing about calling it.

  • You authenticate to the platform, not to the device. A session or an API key on the Authorization header is what the proxy checks, and the credential is consumed there and never forwarded to the device. The capability required is model_serving:invoke, which is deliberately separate from being able to see a deployment: reading a list is not the same as spending a device's GPU.
  • The address exists only while the instance is running. A device's proxy address appears when its instance reports running and is absent otherwise, because an address that cannot answer is worse than no address.
  • It is an operator and integration surface, not a production front door. The platform applies a request-size limit, a per-deployment rate limit and a per-device concurrency cap, and refuses with 429 and a Retry-After rather than queueing without bound. A refusal names its reason: SERVING_NOT_RUNNING when the instance is not up, DEVICE_OFFLINE when the device is gone, SERVING_CONCURRENCY_LIMIT when the device is already busy with as many requests as it will take.

The endpoints a deployment exposes​

The endpoint set depends on what the model does. Every deployment answers the three that describe the server itself:

GET /v1/models      what this server is serving
GET /v1/health      liveness, which the platform polls to decide the instance is up
GET /props          the server's own properties

On top of those, the model's task decides what is meaningful.

The model doesEndpoint
Text generation, including vision-language chatPOST /v1/chat/completions
Speech to textPOST /v1/audio/transcriptions
Embeddings or sentence similarityPOST /v1/embeddings
Text to speechPOST /v1/audio/speech, plus POST /v1/voices to register a voice

The shapes are OpenAI-compatible, so an existing client library usually works by pointing its base URL at the proxy address.

Two details save a confused half hour. Send the model id that GET /v1/models returns as the model value, because the server derives that id from the checkpoint it loaded rather than from the name you gave the deployment. And for a vision model, send images as content parts carrying a data: URI: the server fetches no remote URLs.

The server itself mounts every route and refuses the ones its model cannot serve with an explanatory 400, so a call to the wrong endpoint tells you what happened. Where the platform does not know the model's task, it lists only the three universal endpoints and points you at GET /v1/models and GET /props to see what the server is actually serving.

How ClikaRT does the work​

The process on the device is the CLIKA inference engine, clika-modelverse, running its serve mode with ClikaRT underneath it. The platform stages a small wrapper script as an artifact, starts it as a managed service, and the wrapper finds the engine and hands it the model.

Where the engine comes from, in the order the wrapper looks: an explicit path in the device's environment, then the engine bundle root that benchmark runs already use, then the newest engine bundle staged under the device's staging directory, then whatever is on the path. A device already set up for benchmarking therefore serves with no additional configuration.

The model reaches the device the same way a benchmark's model does. For a Hugging Face model the engine fetches the checkpoint itself, honoring the access token when the repository is gated; for an uploaded model the platform delivers the artifact. A checkpoint already staged on the device is used as it is.

Lifecycle​

A deployment's status is a roll-up of its devices: running when all of them are, partial when some are, failed when none came up, stopped when all were stopped, and pending before the first start. Each device reports its own state (starting, running, failed, stopped) with its process id, port, uptime and restart count, and a failed instance carries its exit code and the tail of what it printed, so a red badge always comes with an explanation.

Auto-restart is on. The device restarts a crashed instance with an exponential backoff that starts at a few seconds and caps at five minutes, gives up after ten consecutive failures and settles into failed, and resets its failure count once an instance has stayed up for a minute. That combination keeps a flapping deployment from restarting forever while letting a long-lived one recover from an occasional crash.

Stopping a deployment removes it from the device's desired state first, so the reconciler does not bring it back, then stops the process group. The configuration is kept, so starting it again re-applies what it had. Deleting it stops every instance and cleans up what the platform staged.

Where you see it​

Deployments in the sidebar lists the project's deployments with their status and devices. Deploy on a benchmark result creates one from the model that run measured. A deployment's own page carries the per-device status, the endpoint list with copyable examples, and the live log per device.

  • Service: the free-form version of the same machinery, for a process the platform does not build for you.
  • Device: what a deployment runs on.
  • Benchmark: how you choose which model to deploy.