Skip to main content

Benchmark a model on this machine

Whether a model meets your latency and throughput targets is a property of this machine, this quantization and this serving shape, and bench measures exactly that. Every runnable text family provides it, the flow drives the same serving pipeline prompt and serve run on, and the result is a structured table, never ad-hoc prints.

The default sweep​

clika-modelverse meta-llama/Llama-3.2-1B-Instruct bench --device cuda

The default sweep runs three serving-shaped cells (balanced 128:128, prefill-heavy 2048:128, decode-heavy 128:1024) over the default concurrency ladder (1, 4, 8), each cell strictly one at a time so cells never contend with each other. One CSV row per cell and concurrency, the last column naming what the row measured:

isl,osl,concurrency,ttft_ms,prefill_tok_s,itl_p50_ms,itl_p99_ms,prefill_chunk_ms_p50,prefill_chunk_ms_max,decode_tok_s,speedup_vs_c1,batching_efficiency,admitted_tok,retires,status,reason,notes
128,128,1,6.864928,18645.498,4.660638,8.971654,0,0,203.36226,1,1,0,2,ok,"",prompts=cold (a fresh prompt every iteration; prefill_tok_s prices prompt ingest)
128,128,4,19.85135,6447.924,5.458697,13.789374,0,0,651.78723,3.2050552,0.8012638,0,8,ok,"",prompts=cold (a fresh prompt every iteration; prefill_tok_s prices prompt ingest)
128,128,8,33.74735,3910.5552,4.789912,12.197703,0,0,1545.349,7.598996,0.9498745,0,16,ok,"",prompts=cold (a fresh prompt every iteration; prefill_tok_s prices prompt ingest)
2048,128,1,61.346237,33384.28,4.935648,9.665019,0,0,178.64566,1,1,0,2,ok,"",prompts=cold (a fresh prompt every iteration; prefill_tok_s prices prompt ingest)
2048,128,4,186.73393,14730.354,6.8231645,27.292572,0,0,478.35248,2.6776607,0.6694152,0,8,ok,"",prompts=cold (a fresh prompt every iteration; prefill_tok_s prices prompt ingest)
2048,128,8,443.7948,4614.745,5.977193,15.1585245,0,0,817.9034,4.578356,0.5722945,0,16,ok,"",prompts=cold (a fresh prompt every iteration; prefill_tok_s prices prompt ingest)
128,1024,1,6.130809,20878.158,4.774493,9.330061,0,0,203.21771,1,1,0,2,ok,"",prompts=cold (a fresh prompt every iteration; prefill_tok_s prices prompt ingest)
128,1024,4,16.072428,8384.906,5.683443,14.419685,0,0,695.1792,3.4208593,0.85521483,0,8,ok,"",prompts=cold (a fresh prompt every iteration; prefill_tok_s prices prompt ingest)
128,1024,8,31.530767,4059.5269,5.0563264,13.9837,0,0,1479.9001,7.282338,0.91029227,0,16,ok,"",prompts=cold (a fresh prompt every iteration; prefill_tok_s prices prompt ingest)

The numbers are one machine's run (an RTX 4080, this model at its checkpoint dtype, bf16, cold prompts, the default); yours are the point. Stdout carries the CSV, ready for a spreadsheet or a script; diagnostics ride stderr. A failed cell reports error in status without invalidating the rows around it, and the reason column carries the model's own refusal or the cell's first failure in words (empty on an ok row). With --output, the report streams to disk as rows complete, so a run killed at its deadline keeps every finished cell.

Shape the sweep to your workload​

  • --cells 512:256,4096:64: your own isl:osl list, replacing the default three. --isl/--osl set a single cell directly.
  • --concurrency 1,4,16: the parallel-session ladder; speedup_vs_c1 and batching_efficiency in the report tell you what the added concurrency actually bought.
  • --shared-prefix 256: how many prompt tokens every session shares (a system prompt, a RAG preamble); with it the admitted_tok and retires columns show the paged prefix cache engaging. Every LLM cell runs cold prompts by default (a fresh random prompt per session every iteration, so a paged prefix cache serves no repeat; --unique-prompts names that default and stays accepted). --repeat-prompts is the warm protocol: the warm-up and every iteration reuse one prompt set, so prefill_tok_s reads cache admits, and the row's notes say so.
  • --warmup and --iters: runs discarded before timing, then runs averaged.
  • The load knobs apply unchanged: --kvcache paged|continuous, --prefill-chunk, --step-token-budget, a GGUF weight selector on the source, --max-seq.
  • --output PREFIX writes the report to files beside the terminal view; --profile adds the per-op profile on stderr (the summary opens with Profiling Report Summary:, its top_ops: block ranks op groups by self time, and the per-op table carries an AvgSelf(us) column, the per-op average that excludes nested spans), --profile-pipeline the per-node one.

Reading the columns​

ttft_ms is the enqueue-to-first-token wall, the number an interactive user feels. prefill_tok_s is prompt ingest; decode_tok_s is the aggregate generation rate across sessions. The inter-token percentiles itl_p50_ms/itl_p99_ms sample decode-only gaps; on a chunked-prefill run the chunk boundary walls report separately (prefill_chunk_ms_p50/_max), so a decode tail is never polluted by an ingest wall. When the p99 sits far above the p50, the deployment story is a batching or cache-pressure story, and the concurrency ladder narrows down which. The -v diagnostics carry the load and peak-memory figures the Model requirements page's estimates are checked against.

Audio and media axes​

A speech-to-text family's bench sweeps audio length instead of token shapes (--audio-seconds 5,30,120, with the same --concurrency ladder), reporting ingest as encoder frames over the end-to-end transcribe wall. Multimodal families add mm_bench, the media sweep over the serving path: --images per request, --image-resolution, --video-frames, with effective_isl reporting what the prefill actually ingested once media expansions counted, so a media-heavy row's ingest rate stays comparable to a text row's.

Benchmark the quantization you intend to ship (Run a specific GGUF quantization); a Q4_K_M and a Q8_0 of the same model can sit on different sides of a latency target. The serving flags that shaped a good bench row carry directly onto serve (Serve an OpenAI-compatible endpoint).