Skip to main content

Benchmark

A benchmark is one run you created: a set of models, a set of quality tests, and a set of devices. The platform executes every combination and returns one result set you can read side by side. The API calls a run a benchmark group, and the web application shows a run as a single row under Benchmarks.

Why it exists​

Measuring a model on a device is a small job. Comparing two models on three devices is bookkeeping, and doing it by hand is where mistakes come from: a different script here, a different prompt set there, and the numbers stop being comparable. A benchmark run makes the comparison the unit of work. Every leg runs the same test with the same harness, and every result carries the model, the device and the test it belongs to.

What you choose, and what the platform chooses​

You choose three things: the models, the quality tests, and the devices. That is the whole input.

The platform chooses how to run each combination. A benchmark type plus a device's platform and architecture resolve to exactly one job definition, through a compatibility table the deployment owns. If a device shape has no entry for that test, the dispatch refuses that device with DEVICE_NOT_RUNNABLE rather than falling back to something unproven. There is no "pick your runner" step, and a benchmark that is confirmed on Linux arm64 but not yet on Android says so instead of producing a number you cannot trust.

Two shapes of benchmark​

A performance benchmark measures how fast a model runs: tokens per second, time to first token, latency, peak memory. A quality benchmark measures whether its answers are right, and each one scores that differently. A single run can carry both.

Quality scoring on this platform is deterministic wherever it can be. Multiple-choice answers are matched exactly against the gold letter, free-text answers by normalized exact match, code by executing it against the benchmark's own tests, translations and summaries by their published overlap metrics. No benchmark uses a model to grade another model.

The catalogue​

What a deployment offers depends on which benchmarks it has installed and which datasets have been staged on it, so treat the list below as the shape of the catalogue rather than a promise about your own platform. GET /api/benchmark-types answers that question for a specific deployment, and the picker only offers the tests whose task matches the models you selected.

The catalogue is grouped the way the platform groups models: by the task family a benchmark applies to. A benchmark is offered for a model when the model's task belongs to its family, which is why an automatic-speech-recognition model is never asked to answer multiple-choice questions.

One property cuts across the families and is worth knowing before you read any quality score. Several benchmarks run a public Hugging Face model through transformers on the device rather than through the ClikaRT engine, which makes them a measurement of the model and not of the engine or its quantization. Each entry says so where it applies.

LLM​

Text-generation models: the family most people mean by "a model". These benchmarks send prompts and read completions, and they split into one performance test and a set of quality tests that score the text that comes back.

How the scores read: the quality numbers here are the ones with the widest published baselines, which makes them tempting to compare against a leaderboard. Several of these datasets are also present in most models' pre-training data, so the honest use is a relative one, comparing the same model across engines, quantizations or devices, rather than an absolute claim about a model's intelligence.

LLM Performance​

Text-generation throughput and latency under a representative inference workload. Reports tokens per second, time to first token and inter-token latency, with peak memory alongside. This is the benchmark to run when the question is how fast a model is on a device rather than how good its answers are.

Accuracy (MMLU)​

Multiple-choice questions across academic and professional subjects, in the style of MMLU. Scored by exact match of the predicted choice letter against the gold answer, with throughput and latency reported alongside. The per-question record shows the subject, the choices in the order the model saw them, and the letter it picked, so a score can be traced to the questions behind it.

GPQA Diamond​

Graduate-level science questions written to be hard for a non-expert with a search engine (198 items in the Diamond subset). Multiple choice, scored by exact match of the choice letter.

ARC Challenge​

Grade-school science questions from the AI2 Reasoning Challenge, challenge set. Multiple choice, scored by exact match. Widely present in pre-training corpora, so read it as a relative yardstick rather than an absolute quality claim.

HellaSwag​

Commonsense sentence completion: pick the ending that plausibly continues a described situation. Multiple choice on the validation split, scored by exact match, with the same caveat about pre-training exposure as ARC.

TruthfulQA mc1​

Resistance to common misconceptions, using the single-correct-answer (mc1) scoring. Scored by exact match of the choice letter. The published mc2 figure is a different metric and is not comparable to this one.

GSM8K​

Grade-school mathematical word problems answered with chain-of-thought reasoning. Scored on numeric equality of the final number, so 1,203, $1203 and 1203.0 all count as the same answer. An answer containing no number at all is counted unparseable rather than wrong, which keeps a model that never answers distinguishable from one that answers badly.

AA-LCR​

Long-context reasoning over multi-document source material, answered as free text and scored by normalized exact match. This is the benchmark for the question "does the model still reason when the prompt is long".

Coding (HumanEval+)​

Functional correctness on HumanEval+ from EvalPlus: the model writes a function, and the platform runs the benchmark's tests against it. Scored by execution, not by resemblance to a reference solution.

Coding (MBPP+)​

The same harness over MBPP+ (399 tasks), a different distribution of programming problems. It is a separate benchmark rather than more items in the same one, because a pass rate mixed across two task distributions is not a comparable number.

Coding (SciCode)​

Scientific-computing code generation, scored by executing the generated program against the benchmark's tests. The tasks assume domain context, so it separates models that can write code from models that can write the right code for a scientific problem.

Spreadsheet (SpreadsheetBench 2)​

Business spreadsheet tasks: the model produces a script, the platform executes it, and the resulting workbook is diffed against the golden one over the cell ranges the task grades. Closer to real office automation than a coding puzzle.

Instruction Following (IFEval)​

Prompts carrying machine-checkable constraints (word counts, forbidden characters, required formats). The answer is scored by running those checkers, with no reference text and no judge model involved. Reported as the strict per-prompt rate, where a prompt counts only when every one of its instructions is satisfied, with the per-instruction rate alongside.

Summarization (CNN/DailyMail, ROUGE)​

News-article summarization scored by ROUGE against the reference highlights, with ROUGE-L F1 as the headline number. ROUGE measures n-gram overlap rather than meaning, so a correct summary worded differently scores low. Read it as a regression yardstick on one model, not as a summarization-quality claim.

Multimodal​

Vision-language models: the ones that take an image (or sampled video frames) alongside text and answer in text. The task is image-text-to-text or document question answering, and the model under test is still a chat model, which is why these read like the LLM quality tests with a picture attached.

How the scores read: every benchmark in this family runs a public Hugging Face model through transformers on the device rather than through the ClikaRT engine, so a score measures the model, not the engine or its quantization. Free-text answers are scored by normalized exact match, which is strict about wording, so a lower number here does not always mean a worse answer.

MMMU multiple-choice​

College-level, multi-discipline questions over images (MMMU validation split, multiple-choice rows). Scored by exact match of the choice letter.

MMMU-Pro​

The harder MMMU-Pro variant, standard four-option split. A separate benchmark from MMMU, and the two numbers are not comparable.

CharXiv reasoning​

Chart-understanding questions over scientific figures, answered as free text and scored by normalized exact match. The question is whether the model can read a plot rather than recognize an object.

MathVision​

Visual mathematics over figure images (testmini, open-ended rows), answered as free text and scored by normalized exact match.

ZeroBench​

Deliberately hard multimodal reasoning, 100 items, one greedy sample per item. Free text, normalized exact match. The published figure for this dataset is a pass rate over five samples, so a single-sample score here reads lower by construction.

BabyVision free-response​

Fine-grained visual discrimination, free-response rows only, scored by normalized exact match.

DocVQA​

Document visual question answering: read the answer out of a scanned page. Scored by ANLS, a normalized edit-distance similarity against each acceptable answer, zeroed below a threshold so an OCR near-miss keeps partial credit while a wrong answer earns none.

OmniDocBench page OCR​

Whole-page document transcription to markdown over annotated PDF pages. The device produces one transcription per page and a scorer compares it against ground truth covering text, formulas and tables in reading order, reporting a mean page-level normalized edit distance where lower is better. It needs server-side scoring enabled on the deployment.

VQAv2​

Open-ended visual question answering scored by the official consensus rule: a prediction agreeing with three or more of the ten human annotators earns full credit, one or two earns partial.

WorldVQA​

Real-world visual question answering over staged images, free text, normalized exact match.

Vision​

Classifiers, detectors and segmenters: models whose output is a label, a box or a mask rather than text. There is no prompt and no completion here, which is why these benchmarks report the metrics computer vision uses rather than an accuracy over answers.

How the scores read: these also run a public Hugging Face model through transformers on the device rather than through the ClikaRT engine. Preprocessing follows each checkpoint's own image processor rather than a fixed recipe, so a figure is a reliable yardstick for the same model across devices or quantizations and a poor one for ranking two publishers' models against each other. Several of the datasets are licensed for non-commercial research only, and a deployment has to stage them before the benchmark can run.

ImageNet-1k​

Image classification over the ILSVRC-2012 validation split, reporting top-1 and top-5 accuracy. The dataset is account-gated and restricted to non-commercial research and education.

COCO detection​

Object detection over the COCO 2017 validation split (80 classes), scored by mean average precision at IoU .5 to .95, with mAP@50 and mAP@75 alongside.

ADE20K segmentation​

Semantic segmentation over ADE20K scene parsing, scored by mean intersection-over-union with pixel accuracy alongside. Released for non-commercial research and educational use.

Cityscapes segmentation​

Semantic segmentation over urban street scenes, same metrics as ADE20K. Licensed for non-commercial use, and its canonical distribution requires registration.

Image Classification (top-1)​

Top-1 accuracy against a labeled image set through the ClikaRT engine. Not runnable today: no engine classifier verb exists yet, so no device shape matches it and a dispatch refuses rather than substituting something else.

Audio​

Speech and audio models: transcription, voice generation and audio-event classification. Two of these run through the ClikaRT engine, which makes them the audio family's answer to both questions at once, how fast and how good.

How the scores read: the transcription metrics are error rates, so lower is better, which inverts the reading of every other quality number in the catalogue. Speed is reported as a real-time factor, the seconds of audio processed per second of wall clock, so a figure above one means faster than real time.

Speech-to-Text (WER)​

Speech-to-text accuracy on read speech, scored by word error rate and character error rate. Real-time factor and audio throughput are reported alongside, and the per-clip record carries the audio, the reference and a word-level diff against the prediction.

STT Performance​

Speech-to-text throughput and latency alone: the real-time factor and audio-seconds per second under a representative workload, with no accuracy scoring.

Text-to-Speech (voice generation)​

Voice generation from a fixed sentence set through the ClikaRT engine. Each generated clip is captured with its prompt, and the run reports real-time factor, latency and throughput plus a round-trip word error rate obtained by transcribing the generated audio back to text.

AudioSet tagging​

Multi-label audio-event classification over AudioSet's sound-event classes, scored by mean average precision. The model under test is an audio classifier, and the score is computed over the whole run rather than per item, so the number of classes actually present in the subset is reported beside it: a high score over few classes is an easy ranking rather than a good model. Runs through transformers on the device.

Image generation​

Text-to-image models. The output is a picture, so there is no answer to mark right or wrong and the score measures whether the image matches the prompt.

Image Generation (diffusion)​

Text-to-image generation from a fixed prompt set through the ClikaRT engine. Each image is captured with its prompt, and the run reports a CLIP-cosine alignment score between prompt and image alongside images per second and latency.

Embedding​

Embedding models: the output is a vector, and quality is a question about ranking rather than about generated text. These benchmarks embed a corpus and a query set and score how well the ranking puts the right documents first.

How the scores read: nDCG@10 is reported as a 0 to 1 ratio, so a published two-digit figure is this value times 100. Both datasets carry usage restrictions, and both run a public Hugging Face model through transformers on the device rather than through the ClikaRT engine.

BEIR NFCorpus​

Zero-shot medical information retrieval over a document corpus and query set, scored by nDCG@10 with Recall@10 and Recall@100 alongside.

BEIR SciFact​

Zero-shot scientific-claim retrieval, same metrics and reporting as NFCorpus. The upstream dataset is non-commercial.

Reading a run​

A completed run gives you the headline metrics per model and device, the per-item record behind each score, and the run's own metadata. Read the results walks the result view, and Read benchmark results covers pulling the same numbers from the CLI or an assistant.

Share Results on a run mints a link that shows it to someone without an account. There is one live link per run, and revoking it invalidates that link immediately.

Where you see it​

Benchmarks in the sidebar lists your runs. New Benchmark opens the picker: models, then quality tests, then devices. A run's own page carries the summary, the per-metric comparison and the per-item output viewer.

  • Job: what one leg of a run does on the device.
  • Artifact: the datasets and engine bundles a run stages.
  • Run a benchmark in the tutorial.