--- title: Model deployment description: "Running one of your models on your own devices as an endpoint: the platform proxy, the endpoints a served model exposes, the deployment page, and how ClikaRT does the work." --- import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; A model deployment runs one of your registered models on one or more of your devices and puts an HTTP endpoint in front of it. The model loads once and stays loaded, so requests are answered without paying the load cost each time. The web application calls this surface **Deployments**, and the **Deploy** button on a benchmark result leads to it. In the API the object is a **serving**. ## Why it exists A benchmark tells you how a model performs on a device. A deployment is how you then use it, with an application, a notebook or a colleague sending requests and the model answering from the hardware you chose. It is also the shortest path from "this model won the benchmark" to "this model is answering", because the platform builds the whole service for you: it stages the runner, starts the process, watches its health, restarts it when it dies, and gives you an address to call. ## What you choose A deployment takes a model, the devices to run it on, and how much of each device to use. | Choice | What it does | | --- | --- | | Model | Any model registered in the project. | | Devices | One or more. Each device runs its own copy of the model and answers on its own endpoint. | | Compute | `auto`, `cpu` or a GPU. New deployments run `auto`, which lets the engine place the model, and the deployment records where it landed; a deployment created before that default keeps its pinned value. | | Serve settings | The context window, the number of parallel requests and the engine's other serve parameters. The platform derives them from the device's memory and the model, shows each with its basis in the **Configure** dialog of the deployment's page, and lets you override any of them there; **Save** keeps the change for the next start, **Save and restart** applies it now, and a saved-but-not-applied change shows as restart pending under the page's header. | A deployment is admitted only where it fits: a start on a GPU whose memory cannot hold the model's weights, its cache and the engine's working reserve is refused with the arithmetic that says what holds the card; a device whose free disk cannot take the weights is refused before anything downloads; and a device whose platform cannot host a deployment is refused at creation, with the reason shown in the picker. Deploying from the web application means create and start in one action. Through the API the two are separate: a serving is created `pending`, and a start call brings it up. ## The proxy The model server on a device binds the device's loopback interface, so nothing outside the device can reach it directly. Every call goes through the platform instead, at a per-device address: ```text /api/v1/servings/{serving_id}/devices/{device_id}/proxy/{path} ``` Everything after `proxy/` is passed through to the model server verbatim, the query string travels with it, and responses stream rather than being buffered, so token-by-token output arrives as it is produced. Three things are worth knowing about calling it. - **You authenticate to the platform, not to the device.** A session or an API key on the `Authorization` header is what the proxy checks, and the credential is consumed there and never forwarded to the device. The capability required is `model_serving:invoke`, which is deliberately separate from being able to see a deployment: reading a list is not the same as spending a device's GPU. - **The address exists only while the instance is running.** A device's proxy address appears when its instance reports running and is absent otherwise, because an address that cannot answer is worse than no address. - **It is an operator and integration surface, not a production front door.** The platform applies a request-size limit, a per-deployment rate limit and a per-device concurrency cap, and refuses with `429` and a `Retry-After` rather than queueing without bound. A refusal names its reason: `SERVING_NOT_RUNNING` when the instance is not up, `DEVICE_OFFLINE` when the device is gone, `SERVING_CONCURRENCY_LIMIT` when the device is already busy with as many requests as it will take. ## The endpoints a deployment exposes The endpoint set depends on what the model does. Every deployment answers the three that describe the server itself: ```text GET /v1/models what this server is serving GET /v1/health liveness, which the platform polls to decide the instance is up GET /props the server's own properties ``` On top of those, the model's task decides what is meaningful. | The model does | Endpoint | | --- | --- | | Text generation, including vision-language chat | `POST /v1/chat/completions` | | Speech to text | `POST /v1/audio/transcriptions` | | Embeddings or sentence similarity | `POST /v1/embeddings` | | Text to speech | `POST /v1/audio/speech`, plus `POST /v1/voices` to register a voice | The shapes are OpenAI-compatible, so an existing client library usually works by pointing its base URL at the proxy address. Two details save a confused half hour. Send the model id that `GET /v1/models` returns as the `model` value, because the server derives that id from the checkpoint it loaded rather than from the name you gave the deployment. And for a vision model, send images as content parts carrying a `data:` URI: the server fetches no remote URLs. The server itself mounts every route and refuses the ones its model cannot serve with an explanatory `400`, so a call to the wrong endpoint tells you what happened. Where the platform does not know the model's task, it lists only the three universal endpoints and points you at `GET /v1/models` and `GET /props` to see what the server is actually serving. ## How ClikaRT does the work The process on the device is the CLIKA inference engine, `clikart-cli`, running its serve mode with ClikaRT underneath it. The platform stages a small wrapper script as an artifact, starts it as a managed service, and the wrapper finds the engine and hands it the model. Where the engine comes from, in the order the wrapper looks: an explicit path in the device's environment, then the engine bundle root that benchmark runs already use, then the newest engine bundle staged under the device's staging directory, then whatever is on the path. A device already set up for benchmarking therefore serves with no additional configuration. The model reaches the device the same way a benchmark's model does. For a Hugging Face model the engine fetches the checkpoint itself, honoring the [access token](../getting-started/first-benchmark/04-add-a-model.md#the-hugging-face-access-token) when the repository is gated; for an uploaded model the platform delivers the artifact. A checkpoint already staged on the device is used as it is. ### The engine's own license Each instance of a deployment runs the engine under a runtime credential the platform mints for it when the instance starts and revokes when it stops; when the device reconnects after losing the platform, the agent fetches a fresh one and restarts the engine under it. You never issue this credential yourself, and it does not count against the credentials you mint for your own applications. The agent launches the engine directly, so a device needs no Python or shell for a deployment. ## Lifecycle A deployment's status is a roll-up of its devices: `running` when all of them are, `partial` when some are, `failed` when none came up, `stopped` when all were stopped, and `pending` before the first start. Each device reports its own state (`starting`, `running`, `failed`, `stopped`) with its process id, port, uptime and restart count, and a failed instance carries its exit code and the tail of what it printed, so a red badge always comes with an explanation. Auto-restart is on. The device restarts a crashed instance with an exponential backoff that starts at a few seconds and caps at five minutes, gives up after ten consecutive failures and settles into `failed`, and resets its failure count once an instance has stayed up for a minute. That combination keeps a flapping deployment from restarting forever while letting a long-lived one recover from an occasional crash. Stopping a deployment removes it from the device's desired state first, so the reconciler does not bring it back, then stops the process group. The configuration is kept, so starting it again re-applies what it had. Deleting it stops every instance and cleans up what the platform staged. ## The deployment page A deployment's page replaces the Deployments list in place and reads top to bottom in the order an operator needs it. **Header.** A back link to the Deployments list (or to the model the page was opened from), then the deployment's name, renamed in place from the pencil beside it, with its status and when the serve last reported. To the right, **Configure** opens the serve settings dialog; **Stop** or **Start** and **Restart** act on every device; **Open inference UI** opens the engine's own chat page and dashboard in a new tab, through the platform proxy. The last one needs the `model_serving:invoke` capability and a running instance; with several running devices it asks which one. The kebab at the top right holds **Delete**, with the choice of also removing the model's files from the devices where no other deployment uses them. A saved serve setting that is not applied yet shows as a restart-pending line under the header. **What runs where.** A card for the model and one for each device, each a shortcut to its own page. A device card carries the instance's uptime and the device's live readings (CPU, memory, disk and temperature) as on the device page; a device that is not reporting says so with when it was last seen. Where the engine placed the model is on the card's hover. A failed or restarted instance adds a line under the cards with its state, its restart count, the failure in one sentence and the engine's log behind a disclosure. A model whose checkpoint carries no chat template is noted here too, because its chat endpoint continues text rather than answering. **Server logs.** The inference server's output, one viewer per device, following the tail; **Pause Auto-scroll** holds the view while you read. **API.** The address to call the deployment at, one per device that is serving, through the platform (works from anywhere that reaches the platform, with an API key), and under a rule the device's own address on its network. Where the server binds the device's loopback interface, which is the default, that address answers only from the device itself, and the card says so. **View API reference** opens the endpoint list for the model's task: each row is a method, a path and a description, and opens to the full request URL, a request and response example, the streaming shape where the endpoint streams, and a `curl` you can paste after exporting your API key. ## Where you see it
**Deployments** in the sidebar lists the project's deployments with their status and devices. **Deploy** on a benchmark result creates one from the model that run measured. A deployment's own page is described in [The deployment page](#the-deployment-page).
```bash clika-cli servings list # the project's deployments clika-cli servings get # one deployment, with per-device status clika-cli servings start # start it; stop and restart alongside clika-cli servings devices-proxy /v1/models ``` The last one calls a running deployment through the platform proxy, which is the quickest way to check what a server is actually serving. See the [Using the CLI](../cli/index.md).
Ask Claude: > Which models are deployed right now, and on which devices? That reaches `get_v1_servings` and `get_v1_servings_id`. Starting or stopping a deployment stays a deliberate action in the web application or the CLI.
## Related pages - [Service](service.mdx): the free-form version of the same machinery, for a process the platform does not build for you. - [Device](device.mdx): what a deployment runs on. - [Benchmark](benchmark.mdx): how you choose which model to deploy.