Skip to main content

Run a benchmark

Everything so far was setup. This part runs the benchmark.

Start a run​

Open Benchmarks and select New Benchmark.

The Benchmarks list with the New Benchmark action

A run is built from three choices, in three sections, and nothing else is asked of you.

Models​

Pick one or more registered models. The tabs on the left filter by category, and Add a Model registers a new one without leaving the dialog.

The models step of the New Benchmark dialog, with models grouped by category

Pick one for a first run. Two models make the result a comparison, which is what the results view is built for, but one is enough to see the loop.

Give the run a name while you are here: the pencil next to "Untitled Benchmark" at the top. A named run is much easier to find a week later.

Quality tests​

Next, choose what to measure. Tests are grouped by category, and only the tests that match your models' tasks are offered.

The quality tests step, with tests grouped into categories

For a first run, Performance is the fastest way to a result: it measures throughput, time to first token, latency and peak memory, and needs no dataset download. Accuracy (MMLU) under Other is a good second choice, and produces per-question data you can browse in the next part.

See Benchmark for the whole catalogue: what each benchmark is, what it tests, and how it is scored.

Selecting a category selects the tests inside it. Selecting several tests runs several tests, so a run over two models, two tests and two devices is eight jobs.

Devices​

Last, choose where to run.

The devices step, listing hardware, operating system, architecture and status, with Run Benchmark at the bottom

The list shows the same hardware facts as the device page, so you can pick by what the machine is rather than by its name. Offline devices are visible but cannot take work.

Then select Run Benchmark.

Watch it​

The run appears at the top of Benchmarks with a live status. Open it while it is going and the page shows what is happening rather than making you wait for a verdict.

A running benchmark: the leg's status, the device's live health, the finished count and a live log

Each leg of the run is a row: the model, the device, and the device's own health while it works, so a run that is slow because the machine is saturated says so. Underneath, a progress line counts the finished legs, and the live log streams what the benchmark is printing on the device.

How long it takes is mostly the device's business. A small model answering a few hundred questions on a workstation is minutes. The first run against a given engine on a given device is slower than the ones after it, because the engine bundle transfers once and is then recognized by checksum.

What you have now​

A run with results attached. The next part reads them.