Skip to main content

Read benchmark results

A finished run carries more than a score. This guide covers what the numbers mean, how to get them out of the platform programmatically, and what to do with a leg that did not finish.

Reading a run in the web application is covered by the tutorial: Read the results.

What the metrics mean​

MetricWhat it measuresDirection
tokens_per_secDecode throughput: tokens produced per second.Higher is better.
ttft_msTime to first token: the wait before output starts.Lower is better.
latency_msEnd to end time for one request.Lower is better.
tpot_msTime per output token, derived from latency, time to first token and the token count.Lower is better.
peak_memory_mbThe most memory the run held at once.Lower is better.
rtfReal-time factor, for speech models: audio seconds processed per second.Higher is better.
A quality scoreWhat the test defines: accuracy, pass rate, ROUGE, BLEU, word error rate.Stated per test.

Where a metric was measured per sample, the summary also carries <metric>_min, <metric>_median and <metric>_max. A metric with no per-sample spread (peak memory, a single accuracy score) stays a point value, and the result view shows the same number for all three rather than inventing a range.

Not every run carries every key. A run whose engine reported no time to first token omits that family instead of reporting a zero.

Get the results from the API​

Start from the run (the API calls it a benchmark group) and walk down to a job.

# the run, with its legs
curl -H "Authorization: Bearer $CLIKA_API_KEY" \
  https://platform.clika.io/api/benchmark-groups/$GROUP_ID

# the results of every leg
curl -H "Authorization: Bearer $CLIKA_API_KEY" \
  https://platform.clika.io/api/benchmark-groups/$GROUP_ID/results

# one leg's jobs, when you need their ids
curl -H "Authorization: Bearer $CLIKA_API_KEY" \
  https://platform.clika.io/api/benchmark-groups/$GROUP_ID/jobs

With a job id, four routes go progressively deeper.

RouteWhat it returns
GET /api/jobs/{id}/benchmark-resultsThe whole result: summary, metadata, and what the runner recorded about the run.
GET /api/jobs/{id}/benchmark-results/summaryThe headline metrics alone.
GET /api/jobs/{id}/benchmark-results/samplesThe per-sample rows, paginated.
GET /api/jobs/{id}/benchmark-results/ioThe per-sample inputs and outputs, with the io_schema that says how to read them.

The io route is the one behind View output data: each row carries what the model was given, what it answered, what was expected, and whether it counted as correct. The io_schema names the shape (multiple choice, transcription, code generation, and others), so a client can render the fields rather than guessing at them.

The raw file the script wrote is stored as an artifact and stays downloadable from the job, whether or not it parsed.

From the CLI​

# watch a run to completion; exits non-zero if any leg ends other than completed
clika-rt benchmarks watch "nightly-llm-sweep"

# the finished run's results as a table, or as JSON for a script
clika-rt benchmarks results "nightly-llm-sweep"
clika-rt benchmarks results "nightly-llm-sweep" -o json

The non-zero exit on failure is what makes watch usable as a CI gate: dispatch the run, watch it, and let the step fail when a leg does.

From Claude​

With the benchmark-ops toolset connected (Set up MCP for Claude), the same reading is a question:

Read the results of my last benchmark run and tell me which model was fastest and whether any leg failed.

That reaches get_benchmark_groups, get_benchmark_groups_id_results and get_jobs_id_benchmark_results. The per-item evidence is get_jobs_id_benchmark_results_samples, which is worth asking for by name when you want to know what a model got wrong rather than only its score.

Diagnose a leg that did not finish​

A run's Details tab (and GET /api/benchmark-groups/{id}/jobs) gives each leg's state. What the state tells you:

StateWhat happened, and what to look at
failedA step failed, or the declared output file was never written. The job's error detail names the failing step; for a script failure it carries the tail of what the script printed.
partialThe benchmark itself succeeded and the results are usable, but teardown failed. The device is flagged dirty and takes no further benchmark jobs until it is cleared.
interruptedThe device disconnected mid-run. Teardown is retried when it reconnects, and a late result can still complete the job.
Refused at dispatchDEVICE_NOT_RUNNABLE means no confirmed mapping exists for that test on that device's platform and architecture. RESOURCE_MISMATCH names the hardware requirement the device did not meet.

A job that failed with an output-missing flag is saying something specific: collection ran and found nothing at the path the definition declared. That is a job-definition problem (the script wrote elsewhere, or exited before writing) rather than a platform one. The other case is a job that completed carrying a parse diagnostic, which means something was written and it was not a result envelope. Write a job definition covers both contracts.

Share a run​

Share Results on the run's page mints a link that shows it to someone with no account on the deployment. There is one live link per run, and revoking it invalidates that link immediately. Use it for a result you want a customer or a colleague to read, and revoke it when the conversation is over.