Run an ONNX model
You have a model as an .onnx file and want ClikaRT to run it. Three types carry the whole story. io::OnnxModel is the format-level object, the ONNX graph as data: open it, inspect it, edit it, save it. compile() is the one crossing into the executable world, where every node parses into its runtime operator, weights bind, and shapes resolve. The result is a ModelGraph as built. optimize() runs the graph optimizer when you want it, finalize() places the graph and packs its weights, and the finalized graph has one run.
The programs below first open an existing file and read its contract, then build a tiny model from scratch so the page runs with no file on disk, then compile and run it. Any .onnx file works in the first section; nothing depends on the architecture.
Open a model and read its IO contract
Opening is cheap and does not compile anything: you get the graph as data. The IO contract (names, dtypes, dims, with dynamic dims reading as named placeholders or -1) is what you need to prepare feeds; print it before anything else when a model is new to you.
- C++
- Python
int main(int argc, char** argv) {
const std::string path = argc > 1 ? argv[1] : tiny_model_path();
OnnxModel model = OnnxModel::open(path);
std::printf("nodes: %zu initializers: %zu opset: %lld\n",
model.num_nodes(), model.num_initializers(),
static_cast<long long>(model.opset()));
// Hold the returned vectors: a range-for over a temporary would iterate
// storage that is gone before the first step.
const std::vector<ClikaRT::spec::TensorSpec> inputs = model.inputs();
const std::vector<ClikaRT::spec::TensorSpec> outputs = model.outputs();
for (const auto& s : inputs)
std::printf(" input %s\n", s.name.c_str());
for (const auto& s : outputs)
std::printf(" output %s\n", s.name.c_str());
return 0;
}
model = crt.io.OnnxModel.open(sys.argv[1] if len(sys.argv) > 1 else tiny_model_path())
print(f"nodes: {model.num_nodes} initializers: {model.num_initializers} "
f"opset: {model.opset}")
for name, dtype, dims in model.inputs():
print(f" input {name} {list(dims)}") # a dynamic dim reads as -1
for name, dtype, dims in model.outputs():
print(f" output {name} {list(dims)}")
The program opens the path on its command line; with none, tiny_model_path() builds the MLP of the next section into a temporary file, so the page runs with no file on disk.
Build a graph from scratch
When there is no file yet (a test, a fixture, a tool that emits ONNX), the same object builds a graph node by node: declare inputs, add initializers (weights enter as ordinary tensors), add nodes by operator type, name the outputs. The tiny MLP here is y = relu(X W + B) with a dynamic batch dimension.
- C++
- Python
OnnxModel build_tiny_mlp() {
Dim batch; // one Dim object: every use is the SAME dynamic dimension
OnnxModel model = OnnxModel::create("tiny_mlp", /*opset_version=*/21);
model.add_input("X", DataType::Float32, {batch, 4});
const std::string w = model.add_initializer(
Tensor::full({4, 3}, 0.5, DataType::Float32));
const std::string b = model.add_initializer(
Tensor::full({3}, 0.25, DataType::Float32));
const std::vector<std::string> mm = model.add_node("MatMul", {"X", w}, 1);
const std::vector<std::string> sum = model.add_node("Add", {mm[0], b}, 1);
const std::vector<std::string> y = model.add_node("Relu", {sum[0]}, 1);
model.add_output(y[0], DataType::Float32, {batch, 3});
return model;
}
def build_tiny_mlp() -> "crt.io.OnnxModel":
model = crt.io.OnnxModel.create("tiny_mlp", opset_version=21)
model.add_input("X", crt.float32, [-1, 4]) # any value <= 0 is dynamic
w = model.add_initializer(crt.tensor(np.full((4, 3), 0.5, dtype=np.float32)))
b = model.add_initializer(crt.tensor(np.full((3,), 0.25, dtype=np.float32)))
(mm,) = model.add_node("MatMul", ["X", w])
(summed,) = model.add_node("Add", [mm, b])
(out,) = model.add_node("Relu", [summed])
model.add_output(out, crt.float32, [-1, 3])
return model
save(path) writes the graph; open(path) round-trips it structure-intact, and optimize() runs the rewrite pipeline to a fixed point (a minimal graph survives unchanged).
Compile and run
compile() can fail like any load of real weights and shapes, so it returns through the error contract; branch on the code name. It returns the graph as built. optimize() runs the graph optimizer when you want it, and finalize() readies the graph to run (Optimize and finalize covers both). The finalized ModelGraph runs positionally (one tensor per input_names() entry, in that order) and, in Python, also by name.
- C++
- Python
int run(ClikaRT::io::OnnxModel& model) {
ClikaRT::Result<ModelGraph> compiled = CLIKART_TRY(model.compile());
if (!compiled.ok()) {
std::printf("compile: FAILED [%s]\n", compiled.code_name().c_str());
return 1;
}
ModelGraph graph = std::move(compiled.value());
graph.optimize(); // the graph optimizer, when wanted
graph.finalize(); // only a finalized graph runs
const std::vector<float> x = {1.0F, 2.0F, 3.0F, 4.0F, -1.0F, 0.5F, 2.0F, -2.0F};
std::vector<Tensor> outputs = graph.run(
{Tensor::from_data(x.data(), {2, 4}, DataType::Float32)});
std::printf("y = %s\n", outputs[0].to_string().c_str());
return 0;
}
graph = model.compile()
graph.optimize() # the graph optimizer, when wanted
graph.finalize() # only a finalized graph runs
print(f"inputs {graph.input_names()} -> outputs {graph.output_names()}")
x = crt.tensor(np.array([[1, 2, 3, 4], [-1, 0.5, 2, -2]], dtype=np.float32))
(y,) = graph.run([x]) # positional: input_names() order
(y2,) = graph.run({"X": x}) # named: the same result
print(y.numpy())
codes = [n.op_code for n in graph.nodes()] # the compiled graph as data
assert crt.graph.OpCode.MatMul in codes
# The one-call loader: open, compile, optimize and finalize, then call the model with named inputs.
model.save("tiny_mlp.onnx")
loaded = crt.onnx.load("tiny_mlp.onnx", dynamic_axes={"X": {0: "batch"}}, device="cpu")
out = loaded(X=x) # a dict keyed by output name
assert list(out) == loaded.output_names
crt.onnx.load opens, compiles, optimizes and finalizes in one call: dynamic_axes names the axes that vary between calls, device places the weights, and the returned model takes its inputs by keyword and returns a dict keyed by output name. optimize=False returns the graph as built, which runs after model.graph.finalize(). graph.nodes() reads the compiled graph as data, one crt.graph.OpCode per node; Trace eager code to graphs walks that surface.
With the fixed all-half weights and the 0.25 bias above, the math fits in your head; the C++ program's run over the two-row input prints:
y = relu(X W + B):
[5.25, 5.25, 5.25]
[0, 0, 0]
The first row is (1+2+3+4) * 0.5 + 0.25 = 5.25 per output; the second row's pre-activation is negative in every column, so relu zeroes it.
Two pointers from here. A compiled ModelGraph is the same type that tracing eager code produces, so everything downstream of compile() is shared, optimizing and finalizing included. And a model too big to build by hand arrives as a file: the IO-contract section works unchanged on a checkpoint you downloaded.