Run a benchmark

Everything so far was setup. This part runs the benchmark.

Start a run

Open Benchmarks and select New Benchmark.

A run is built in two steps. The first step is the selection: the devices to run on and the models to run, side by side. The second step is the evaluation: the quality tests to run over every model and device pair, and how much of each test's dataset to cover. Nothing else is asked of you.

Give the run a name while you are here: the pencil next to "Untitled benchmark" at the top. A named run is much easier to find a week later.

Step 1: devices and models

The selection step shows two columns, Devices on the left and My Models on the right, and takes them in either order.

Devices lists the devices that can take a run right now, the most recently seen first, with the same hardware facts as the device page. A device that is offline or suspended is not offered. Search, the tag filter and the hardware filter narrow the list, Select all takes everything listed, and Register a device opens the enrollment dialog without leaving the flow; a device registered from there comes back selected once it is online. Where the deployment hosts phones in the cloud, a Cloud hosted devices group sits under your own fleet.

My Models lists the project's registered models, with a type filter and a search box. Add Model registers a Hugging Face model without leaving the dialog, and it comes back selected. A run uses one model type: once a model is selected, models of another type grey out until the selection is cleared.

Each column reads the other. With a device selected, the model list sorts by fit (fits every selected device, then some, then none) and a model that would be skipped somewhere says on which device and why, memory most often. With a model selected, a device row that cannot take every selected model says how many it would skip. Nothing is hidden for its fit; a pair that does not fit is skipped at launch rather than refused.

Pick one device and one model for a first run. Two models make the result a comparison, which is what the results view is built for, but one is enough to see the loop.

The footer counts the pairs (models × devices = runs) and says how many would be skipped. Continue moves to the second step once there is at least one device, one model and one pair that can run.

Step 2: quality tests and dataset

The evaluation step opens with a recap of the selection: the arithmetic of the pairs, how many launch and how many are skipped, and a Show skipped disclosure that names the skipped pairs grouped by model.

Under it, the Quality tests card offers the tests the selected models' tasks can run, grouped by category, each a chip you toggle. A chip reads the test's name, an info button with its description, the license its dataset is used under and where the dataset comes from. A selected test that takes only part of the selection says so on the chip ("runs 3 of 4"), because a test accepts only the models whose task it declares compatible and the pairs that fit it.

For a first run, LLM Performance, under Performance, is the fastest way to a result: it is the one test that measures speed (throughput, time to first token, latency and peak memory), and it needs no dataset download. An accuracy test such as MMLU or ARC-Challenge is a good second choice, and produces per-question data you can browse in the next part.

See Benchmark for the whole catalog: what each test is, what it tests, and how it is scored.

Dataset is one choice for the whole run: Quick covers 10% of each selected test's dataset, at most 100 samples; Full covers the complete datasets. Quick is the default and the right choice for a first run.

Selecting several tests runs several tests, so a run over two models, two tests and two devices is eight jobs. The footer says how many runs will start and how many are skipped, and names any selected device that reports less memory than a test needs; the run still proceeds there. When the selection needs more than one benchmark group (one group carries one test), the footer names the groups it will create.

Then select Run Benchmarks. Back returns to the selection with everything kept; closing the dialog asks before it discards the draft.

Watch it

The run appears at the top of Benchmarks with a live status. Open it while it is going and the page shows what is happening rather than making you wait for a verdict.

Each leg of the run is a row: the model, the device, and the device's own health while it works, so a run that is slow because the machine is saturated says so. Underneath, a progress line counts the finished legs, which for a one-leg run stays at 0 of 1 until the leg completes, and the live log streams what the benchmark is printing on the device: the log is where a run's progress shows, one line per measured cell.

How long it takes is mostly the device's business. A small model answering a few hundred questions on a workstation is minutes. Each leg begins by sending the engine bundle to the device, which the log reports as pushing_artifacts: on a small machine that is the first four to five minutes of a quick LLM Performance run, which takes about fifteen minutes in all on a two-core laptop. Where your organization's plan meters credits, that transfer is what spends them, and a cancelled run that has started its transfer is charged too; the Subscription card under Settings shows the balance.

What you have now

A run with results attached. The next part reads them.

• Full detail: Benchmark.
• Next: Read the results.