Read benchmark results
A finished run carries more than a score. This guide covers what the numbers mean, how to get them out of the platform programmatically, and what to do with a leg that did not finish.
Reading a run in the web application is covered by the tutorial: Read the results.
What the metrics mean
| Metric | What it measures | Direction |
|---|---|---|
tokens_per_sec | Decode throughput: tokens produced per second. | Higher is better. |
ttft_ms | Time to first token: the wait before output starts. | Lower is better. |
latency_ms | End to end time for one request. | Lower is better. |
tpot_ms | Time per output token, derived from latency, time to first token and the token count. | Lower is better. |
peak_memory_mb | The most memory the run held at once. | Lower is better. |
rtf | Real-time factor, for speech models: audio seconds processed per second. | Higher is better. |
| A quality score | What the test defines: accuracy, pass rate, ROUGE, BLEU, word error rate. | Stated per test. |
Where a metric was measured per sample, the summary also carries <metric>_min, <metric>_median and <metric>_max. A metric with no per-sample spread (peak memory, a single accuracy score) stays a point value, and the result view shows the same number for all three rather than inventing a range.
Not every run carries every key. A run whose engine reported no time to first token omits that family instead of reporting a zero.
Get the results from the API
Start from the run (the API calls it a benchmark group) and walk down to a job.
# the run, with its legs
curl -H "Authorization: Bearer $CLIKA_API_KEY" \
https://platform.clika.io/api/benchmark-groups/$GROUP_ID
# the results of every leg
curl -H "Authorization: Bearer $CLIKA_API_KEY" \
https://platform.clika.io/api/benchmark-groups/$GROUP_ID/results
# one leg's jobs, when you need their ids
curl -H "Authorization: Bearer $CLIKA_API_KEY" \
https://platform.clika.io/api/benchmark-groups/$GROUP_ID/jobs
With a job id, four routes go progressively deeper.
| Route | What it returns |
|---|---|
GET /api/jobs/{id}/benchmark-results | The whole result: summary, metadata, and what the runner recorded about the run. |
GET /api/jobs/{id}/benchmark-results/summary | The headline metrics alone. |
GET /api/jobs/{id}/benchmark-results/samples | The per-sample rows, paginated. |
GET /api/jobs/{id}/benchmark-results/io | The per-sample inputs and outputs, with the io_schema that says how to read them. |
The io route is the one behind View output data: each row carries what the model was given, what it answered, what was expected, and whether it counted as correct. The io_schema names the shape (multiple choice, transcription, code generation, and others), so a client can render the fields rather than guessing at them.
The raw file the script wrote is stored as an artifact and stays downloadable from the job, whether or not it parsed.
From the CLI
# watch a run to completion; exits non-zero if any leg ends other than completed
clika-rt benchmarks watch "nightly-llm-sweep"
# the finished run's results as a table, or as JSON for a script
clika-rt benchmarks results "nightly-llm-sweep"
clika-rt benchmarks results "nightly-llm-sweep" -o json
The non-zero exit on failure is what makes watch usable as a CI gate: dispatch the run, watch it, and let the step fail when a leg does.
From Claude
With the benchmark-ops toolset connected (Set up MCP for Claude), the same reading is a question:
Read the results of my last benchmark run and tell me which model was fastest and whether any leg failed.
That reaches get_benchmark_groups, get_benchmark_groups_id_results and get_jobs_id_benchmark_results. The per-item evidence is get_jobs_id_benchmark_results_samples, which is worth asking for by name when you want to know what a model got wrong rather than only its score.
Diagnose a leg that did not finish
A run's Details tab (and GET /api/benchmark-groups/{id}/jobs) gives each leg's state. What the state tells you:
| State | What happened, and what to look at |
|---|---|
failed | A step failed, or the declared output file was never written. The job's error detail names the failing step; for a script failure it carries the tail of what the script printed. |
partial | The benchmark itself succeeded and the results are usable, but teardown failed. The device is flagged dirty and takes no further benchmark jobs until it is cleared. |
interrupted | The device disconnected mid-run. Teardown is retried when it reconnects, and a late result can still complete the job. |
| Refused at dispatch | DEVICE_NOT_RUNNABLE means no confirmed mapping exists for that test on that device's platform and architecture. RESOURCE_MISMATCH names the hardware requirement the device did not meet. |
A job that failed with an output-missing flag is saying something specific: collection ran and found nothing at the path the definition declared. That is a job-definition problem (the script wrote elsewhere, or exited before writing) rather than a platform one. The other case is a job that completed carrying a parse diagnostic, which means something was written and it was not a result envelope. Write a job definition covers both contracts.
Share a run
Share Results on the run's page mints a link that shows it to someone with no account on the deployment. There is one live link per run, and revoking it invalidates that link immediately. Use it for a result you want a customer or a colleague to read, and revoke it when the conversation is over.
Related pages
- Benchmark: what a run is and what a result contains.
- Job: the states above, in full.
- Write a job definition: the envelope your own benchmark has to emit.