Write a job definition

A job definition is the template a benchmark job executes on a device: what to push, what to install, what to run, where the result lands, and what to clean up afterwards. You write one when you are adding a benchmark a deployment does not carry yet. For an ordinary run you never touch one, because the platform picks the definition itself.

This guide covers the file, the contract your script has to honor, and how to get the file into a deployment.

The document

A definition is a YAML document with kind: JobDefinition. Field names are snake_case, matching the JSON API, and several documents can share a file separated by ---.

kind: JobDefinition
name: "Inference Benchmark (llama.cpp, CPU)"
description: "Throughput on a GGUF model through llama-bench, into the structured envelope."
tags:
  category: "inference"

artifacts:
  - artifact_id: "llamacpp-runtime"        # name or lineage UUID
    dest_path: "/tmp/llamacpp-runtime"
    tag: "latest"                          # resolved to a version at dispatch

setup_steps:
  - type: exec
    command: "python3 -m venv /tmp/llamacpp-runtime/.venv && /tmp/llamacpp-runtime/.venv/bin/pip install -r /tmp/llamacpp-runtime/requirements.txt"

script:
  command:
    - "sh"
    - "-lc"
    - 'mkdir -p /tmp/lc-out && exec /tmp/llamacpp-runtime/.venv/bin/python /tmp/llamacpp-runtime/runner.py bench --model "${CLIKA_MODEL_HF_URL#https://huggingface.co/}" --out "${CLIKA_OUTPUT_PATH}"'
  working_dir: "/tmp/llamacpp-runtime"
  timeout_sec: 1800
  env:
    CLIKA_BENCH_DEVICE: "cpu"
    HF_HOME: "/tmp/clika-model-cache"      # keep the weights cache inside a path cleanup can reach

output_path: "/tmp/lc-out/results.json"
result_type: "structured"

teardown_steps: []

required_resources:
  min_disk_bytes: 10737418240              # 10 GiB

cleanup_policy:
  remove_output: false
  custom_paths:
    - "/tmp/clika-model-cache"

That is the whole shape. The committed definitions a deployment ships live under products/clika-runtime-platform/job-defs/<slug>/config.yaml in the platform repository, one directory per model and device variation, and they are the reference to copy from.

Every key

Key	Required	What it does
kind	yes	Must be JobDefinition.
name	yes	The definition's identity. Applying a document twice with the same name updates the existing definition rather than creating a second one.
description	no	What this benchmark measures. It is what a reader sees in the catalog.
tags	no	Key and value pairs for organizing the catalog. Tags do not decide which definition a run uses.
artifacts[]	no	Files pushed to the device before anything runs.
artifacts[].artifact_id	yes	The artifact by name or by lineage UUID.
artifacts[].dest_path	yes	Where it lands on the device. A directory artifact is extracted here; a file artifact is written here verbatim.
artifacts[].tag	no	Which version the tag resolves to at dispatch (latest by default). A tag that does not exist fails the dispatch.
artifacts[].credential_name	no	The registry credential to authenticate an external (Docker or Git) artifact with.
setup_steps[]	no	Steps run in order after the files land, before the script.
script	yes	The benchmark itself.
script.command	yes	The command as an argument array. A shell one-liner is ["sh", "-lc", "..."].
script.working_dir	no	Working directory on the device.
script.env	no	Environment variables, merged with the ones the platform injects.
script.timeout_sec	no	Wall-clock limit. When it elapses the platform stops the whole process group and reports the timeout as itself, not as an exit code. 0 means no limit.
output_path	yes	Where the script must write its result.
result_type	no	raw keeps the file as an artifact. structured parses it, which is what unlocks metrics, charts and comparison. Default raw.
teardown_steps[]	no	Steps that always run at the end, successful or not.
required_resources	no	Minimum hardware, checked before dispatch.
cleanup_policy	no	What is removed after teardown.

Steps

A step is a type plus a command. exec runs a command and waits for it to exit. start_service starts a managed service and waits for it to report running. wait_healthy polls a URL until it answers.

Two spellings of a command exist in the wild, and both work: the argument array (["bash", "-c", "..."]) and a single shell string. The definitions currently deployed use the single-string form for setup and teardown, so a file captured from a live deployment will look like that.

Teardown steps run after the script whatever happened, including after a cancel or a timeout. If a teardown step fails, the device is flagged dirty and takes no further benchmark jobs until it is cleared, so keep teardown simple and idempotent.

Required resources

required_resources:
  min_cpu_cores: 4
  min_memory_bytes: 8589934592
  min_gpu_vram_bytes: 8589934592
  min_gpu_count: 1
  min_disk_bytes: 10737418240
  accelerators: ["cuda"]
  custom:
    jetpack_version: "5.1"

Every numeric minimum must be met, every listed accelerator must be present, and every custom pair must match what the device reported. A dispatch that fails validation returns RESOURCE_MISMATCH and names each unmet requirement (gpu_vram: need 8.0 GB, have 4.0 GB). Passing force on the dispatch bypasses the numeric checks, for the case where you know better than auto-detection.

Cleanup policy

cleanup_policy:
  remove_artifacts: true      # default
  remove_output: false        # keep it false
  custom_paths:
    - "/tmp/clika-model-cache"

Cleanup runs in a fixed order after the script: your teardown steps, then the pushed artifacts, then custom_paths, then the output path. Three rules are worth following exactly.

• Keep remove_output false. The platform collects the output after teardown, and a cleanup that removes it first loses the result silently.
• Never sweep a path that contains your output_path. Same failure, arrived at from the other direction.
• Anything downloaded at run time needs a path cleanup can reach. A model cache under the default $HOME/.cache/huggingface sits outside the agent's writable roots, so point HF_HOME at a path you also list in custom_paths.

Definitions that push a large engine bundle usually set remove_artifacts: false. The agent skips a transfer whose checksum already matches on the device, and that only helps if the file is still there.

The contract your script honors

The platform injects environment variables at dispatch, on top of script.env.

Variable	What it carries
CLIKA_OUTPUT_PATH	The absolute path the result must be written to (output_path, with ${JOB_ID} expanded).
CLIKA_JOB_ID	This job's id. Echo it into the result's metadata.
CLIKA_MODEL_HF_URL	The canonical Hugging Face URL of the model being benchmarked.
CLIKA_MODEL_SOURCE	huggingface when the run named a Hugging Face model.
CLIKA_MODEL_NAME, CLIKA_MODEL_PATH, CLIKA_MODEL_ARTIFACT_ID	Set where the model reaches the device another way.
CLIKA_HF_TOKEN	A Hugging Face token, when the organization has one configured. Absent for public repositories.
CLIKA_SERVER_URL	The platform's externally reachable URL.
CLIKA_AGENT_BIN	The agent's own executable, for definitions that run a benchmark the agent carries.

Writing ${JOB_ID} into output_path is worth doing: it expands platform-side, so two jobs on the same device never write to the same results file.

With result_type: structured, the file at output_path must be the result envelope:

{
  "summary":  { "tokens_per_sec": 242.0, "ttft_ms": 118.4, "latency_ms": 940.2 },
  "samples":  [ { "input": "...", "output": "...", "correct": true } ],
  "metadata": { "job_id": "...", "benchmark": "llm_performance", "device_placed": "cpu" }
}

summary is what the result view charts, samples is the per-item evidence, and metadata records what actually ran. Where a metric was measured per sample, add its _min, _median and _max on the same base name and the result view renders them.

Two collection outcomes are worth designing for. A declared output_path with nothing at it fails the job, even when the script exited zero, because it is a broken contract with the platform. A file that is not a valid envelope completes the job with a diagnostic, and the raw file stays attached, because the format was your script's choice and its exit code was its own verdict.

Apply it

The CLI reads the YAML and applies it by name, which is create-or-update, so a committed file applied twice does not create a duplicate.

clika-rt apply -f config.yaml

Applying the same document twice is safe, which is what makes a committed definition file the source of truth rather than a snapshot of one.

A definition that exists is not yet reachable from the New Benchmark flow. A benchmark run resolves its definition from a table the deployment owns, keyed by benchmark type, device platform and device architecture, so a platform administrator has to add the row that maps your definition to the device shapes it is confirmed to work on. Until that row exists, a run of that type against that device shape refuses with DEVICE_NOT_RUNNABLE, which is the honest answer: nobody has verified the combination yet. The mapping is managed in the admin area's benchmark catalog.

Two footguns

• Windows definitions take a single-element command. The agent runs a script through cmd /c on Windows, and the platform quotes a multi-element array for a POSIX shell, which cmd cannot parse. Write the whole command as one string in cmd.exe syntax, with %VAR% expansion.
• Keep large files out of the output directory. Output collection has failed in practice because a multi-hundred-megabyte engine library sat in the same directory as the result file. Push the engine somewhere else and point output_path at a small directory of its own.

Related pages

• Job: the states a job moves through and what each guarantees.
• Artifact: versions, tags and checksum deduplication.
• Write a service definition: the sibling document for long-running processes.