--- title: Edit a graph description: "Change a ModelGraph before finalize(): rewire the reads of a value, change what a node reads, rename a node, and edit the graph's inputs and outputs. A refused edit changes nothing; then optimize, finalize and run the edited graph." --- {/* Every block is a program under examples//howto/edit_a_graph/: the first block of each tab is get_a_graph whole, and every later block is its program's docs region. tools/tutorial_check.py runs each program against its recorded output. */} A `ModelGraph` that you trace or compile can be changed before you call `finalize()`. You can move the reads of a value to another value, remove or bypass a node, change what a node reads, rename a node, and add, remove or rename the graph's inputs and outputs. Each edit checks everything before it changes anything, so a refused edit leaves the graph as it was and says what it refused. The edits are available from C++ and Python, under the same names. ## A graph to edit A trace returns the graph as built, with every operator as written and nothing optimized or finalized, so it is ready to edit. The model below computes `y = Relu(x) + Neg(x)`: `x` feeds a Relu and a Neg, and one Add reads both. Every example on this page uses it. `label` names a node by its input name or its operator, `labels` lists a graph's nodes, and `producers` lists the nodes a node reads, one per input port. Edits run before `finalize()`. A finalized graph serves `run()` and refuses every edit with the code name `FAILED_PRECONDITION`; to edit it, trace or compile the model again. ```cpp title="get_a_graph.cpp" #include #include #include #include using ClikaRT::DataType; using ClikaRT::Error; using ClikaRT::Tensor; using ClikaRT::graph::ModelGraph; using ClikaRT::graph::Node; using ClikaRT::graph::NodeKind; using ClikaRT::graph::OpCode; using ClikaRT::graph::Value; namespace ops = ClikaRT::ops; namespace { // y = Relu(x) + Neg(x): x feeds a Relu and a Neg, both read by one Add. std::vector model(const std::vector& inputs) { const Tensor rectified = ops::relu(inputs[0]); const Tensor negated = ops::neg(inputs[0]); return {ops::add(rectified, negated)}; } // A node's label: an input's name, an operator's code. std::string label(const Node& node) { return node.kind() == NodeKind::Input ? node.name() : std::string(ClikaRT::graph::op_code_name(node.op_code())); } // The labels of `nodes`, separated by spaces. std::string labels(const std::vector& nodes) { std::string out; for (const Node& node : nodes) out += (out.empty() ? "" : " ") + label(node); return out; } // The labels of the nodes that produce what `node` reads, one per input port. std::string producers(const Node& node) { std::string out; for (const Value& value : node.inputs()) out += (out.empty() ? "" : " ") + label(*value.producer()); return out; } // A trace returns the graph as built: every operator as written, nothing optimized or finalized. ModelGraph trace_model() { const std::vector signature = {{"x", DataType::Float32, {2, 3}}}; const std::vector outputs = {"y"}; return ClikaRT::graph::trace(model, signature, "edit", outputs); } } // namespace int main() { const ModelGraph graph = trace_model(); const Node add = graph.find_nodes(OpCode::Add).front(); std::printf("%s | %s\n", labels(graph.nodes()).c_str(), producers(add).c_str()); // x Relu Neg Add | Relu Neg std::printf("%s %s\n", graph.output_names().front().c_str(), graph.is_finalized() ? "true" : "false"); // y false // Edits run before finalize(): a finalized graph refuses every one of them. ModelGraph done = trace_model(); done.finalize(); try { done.rename_output("y", "total"); } catch (const Error& error) { std::printf("%s\n", error.code_name().c_str()); // FAILED_PRECONDITION } return 0; } ``` ```python title="get_a_graph.py" import clika_runtime as crt from clika_runtime.graph import NodeKind, OpCode def model(inputs: list[crt.Tensor]) -> list[crt.Tensor]: x = inputs[0] return [crt.relu(x) + crt.neg(x)] # y = Relu(x) + Neg(x) def label(node: crt.graph.Node) -> str: return node.name if node.kind == NodeKind.Input else node.op_code.name def labels(graph: crt.graph.ModelGraph) -> list[str]: return [label(node) for node in graph.nodes()] def producers(node: crt.graph.Node) -> list[str]: return [label(value.producer()) for value in node.inputs] # A trace returns the graph as built: every operator as written, nothing optimized or finalized. graph = crt.trace(model, [crt.TensorSpec("x", crt.float32, [2, 3])], output_names=["y"]).graph (add,) = graph.find_nodes(OpCode.Add) print(labels(graph), producers(add)) # ['x', 'Relu', 'Neg', 'Add'] ['Relu', 'Neg'] print(graph.output_names(), graph.is_finalized()) # ['y'] False # Edits run before finalize(): a finalized graph refuses every one of them. done = crt.trace(model, [crt.TensorSpec("x", crt.float32, [2, 3])], output_names=["y"]).graph done.finalize() try: done.rename_output("y", "total") except crt.InvalidArgumentError as error: print(error.code_name) # FAILED_PRECONDITION ``` Edit a graph when a model needs a change its source does not make: an operator to remove before deployment, a value to expose or feed, or a name a caller binds. Each program on this page is complete and runs on its own. From here on, a block shows the part of its program that follows the opening lines the first block shows (the includes or imports, `model`, `label`, `labels`, `producers` and the trace). ## A refused edit changes nothing Every edit checks its arguments and the graph's rules before it changes anything, so a refused edit leaves the nodes, the inputs, the outputs and the constants as they were. Its message names the call and what it refused. The code name is `INVALID_ARGUMENT` for an argument the edit cannot take, and `FAILED_PRECONDITION` for a finalized graph. In C++ the edit throws a `ClikaRT::Error` (a `Result` carries it in a build without exceptions); in Python it raises `InvalidArgumentError`, whose `code_name` says which. Here `rename_node` refuses an input and names the call that renames one. ```cpp title="refused_edit.cpp" int main() { ModelGraph graph = trace_model(); const std::string before = labels(graph.nodes()); try { graph.rename_node(*graph.node("x"), "features"); // an input is renamed with rename_input() } catch (const Error& error) { std::printf("%s | %s\n", error.code_name().c_str(), error.what()); // INVALID_ARGUMENT | rename_node: 'x' is a graph input; rename it with rename_input() } std::printf("%s %s\n", labels(graph.nodes()) == before ? "true" : "false", graph.input_names().front().c_str()); // true x return 0; } ``` ```python title="refused_edit.py" before = labels(graph) try: graph.rename_node(graph.node("x"), "features") # an input is renamed with rename_input() except crt.InvalidArgumentError as error: print(error.code_name, "|", error) # INVALID_ARGUMENT | rename_node: 'x' is a graph input; rename it with rename_input() print(labels(graph) == before, graph.input_names()) # True ['x'] ``` Branch on the code name and report the message; [Handle errors by code](handle-errors-by-code.mdx) covers the channels every failure carries. ## Rewire the reads of a value `replace_all_uses_with(old_value, replacement)` moves every read of a value to another one: each operator input it feeds, and each graph output it returns, which keeps its name. `remove_node(node)` removes an operator that nothing reads, and refuses one that is still read, naming the reader to rewire or remove first. `bypass_node(node)` hands a node's readers the value on its input (the first input and output, unless you name the ports) and removes the node. A value takes another's place only with the same dtype and dims. ```cpp title="rewire.cpp" int main() { ModelGraph graph = trace_model(); const Node relu = graph.find_nodes(OpCode::Relu).front(); const Node neg = graph.find_nodes(OpCode::Neg).front(); const Node add = graph.find_nodes(OpCode::Add).front(); graph.replace_all_uses_with(*neg.output(0), *graph.node("x")->output(0)); // the Add reads x where it read Neg(x) graph.remove_node(neg); // nothing reads the Neg now std::printf("%s\n", labels(graph.nodes()).c_str()); // x Relu Add graph.bypass_node(relu); // the Add reads the Relu's input, x, and the Relu goes std::printf("%s | %s\n", labels(graph.nodes()).c_str(), producers(add).c_str()); // x Add | x x return 0; } ``` ```python title="rewire.py" (relu,) = graph.find_nodes(OpCode.Relu) (neg,) = graph.find_nodes(OpCode.Neg) (add,) = graph.find_nodes(OpCode.Add) graph.replace_all_uses_with(neg.output(0), graph.node("x").output(0)) # the Add reads x where it read Neg(x) graph.remove_node(neg) # nothing reads the Neg now print(labels(graph)) # ['x', 'Relu', 'Add'] graph.bypass_node(relu) # the Add reads the Relu's input, x, and the Relu goes print(labels(graph), producers(add)) # ['x', 'Add'] ['x', 'x'] ``` Rewire a graph to remove an operator a deployment does not need, or to let readers take a value the graph already computes. ## Change what a node reads `set_input(node, port, value)` gives one input port another value; the port must read a value already. `set_constant_input(node, port, tensor)` binds a new constant holding the tensor's bytes to the port: the graph keeps a handle to the bytes, with no copy, and places them with the graph at `finalize()`. `add_constant(tensor)` makes a constant that nothing reads yet, for `set_input` or `replace_all_uses_with` to wire in, and `finalize()` drops a constant that nothing reads by then. A weight an operator holds itself, such as a compiled MatMul's, stays as it is, and an edit that would replace it refuses. ```cpp title="node_inputs.cpp" int main() { ModelGraph graph = trace_model(); const Node relu = graph.find_nodes(OpCode::Relu).front(); const Node neg = graph.find_nodes(OpCode::Neg).front(); const Node add = graph.find_nodes(OpCode::Add).front(); graph.set_input(add, 1, *relu.output(0)); // port 1 reads Relu(x) in place of Neg(x) graph.remove_node(neg); // which nothing reads now std::printf("%s\n", producers(add).c_str()); // Relu Relu // Port 1 reads a constant holding the tensor's bytes. graph.set_constant_input(add, 1, Tensor::full({2, 3}, 1.0, DataType::Float32)); std::printf("%s %zu\n", add.input(1)->is_constant() ? "true" : "false", graph.constants().size()); // true 1 const Value half = graph.add_constant(Tensor::full({2, 3}, 0.5, DataType::Float32)); // nothing reads it yet graph.set_input(add, 1, half); // port 1 reads it, and the ones, read by nothing, go std::printf("%s %zu\n", *add.input(1) == half ? "true" : "false", graph.constants().size()); // true 1 return 0; } ``` ```python title="node_inputs.py" (relu,) = graph.find_nodes(OpCode.Relu) (neg,) = graph.find_nodes(OpCode.Neg) (add,) = graph.find_nodes(OpCode.Add) graph.set_input(add, 1, relu.output(0)) # port 1 reads Relu(x) in place of Neg(x) graph.remove_node(neg) # which nothing reads now print(producers(add)) # ['Relu', 'Relu'] graph.set_constant_input(add, 1, crt.ones(2, 3)) # port 1 reads a constant holding the tensor's bytes print(add.input(1).is_constant(), len(graph.constants())) # True 1 half = graph.add_constant(crt.full((2, 3), 0.5)) # a constant nothing reads yet graph.set_input(add, 1, half) # port 1 reads it, and the ones, read by nothing, go print(add.input(1) == half, len(graph.constants())) # True 1 ``` Change a node's inputs to feed a constant of your own into the graph, or to point an operator at another producer. ## Rename a node `rename_node(node, name)` renames an operator, and its outputs keep their names; an input is renamed with `rename_input` instead. A view of the old name is gone afterwards, so look the node up again with `node(name)`. ```cpp title="rename.cpp" int main() { ModelGraph graph = trace_model(); graph.rename_node(graph.find_nodes(OpCode::Add).front(), "total"); // views of the old name are gone afterwards const Node total = *graph.node("total"); // so find the node again by its new one std::printf("%s | %s | %s\n", label(total).c_str(), producers(total).c_str(), graph.output_names().front().c_str()); // Add | Relu Neg | y return 0; } ``` ```python title="rename.py" (add,) = graph.find_nodes(OpCode.Add) graph.rename_node(add, "total") # views of the old name are gone afterwards total = graph.node("total") # so find the node again by its new one print(label(total), producers(total), graph.output_names()) # Add ['Relu', 'Neg'] ['y'] ``` Rename nodes to give them the names your own tools and reports use. ## Edit the graph's inputs and outputs `add_input(spec)` adds a graph input with the spec's name, dtype and dims (a dynamic dim takes its size at `run()`) and returns its input node, whose value the other edits take. `add_output(value, name)` returns a value under a new output name, and `remove_output(name)` stops returning one, while the node that computes it stays. `rename_input(name, new_name)` and `rename_output(name, new_name)` rename an input and an output. Input and output names stay unique, and a name a KV cache layer binds stays as it is. ```cpp title="graph_io.cpp" int main() { ModelGraph graph = trace_model(); const Node relu = graph.find_nodes(OpCode::Relu).front(); const Node neg = graph.find_nodes(OpCode::Neg).front(); const Node add = graph.find_nodes(OpCode::Add).front(); const Node bias = graph.add_input({"bias", DataType::Float32, {2, 3}}); // a new input, bound at run() graph.set_input(add, 1, *bias.output(0)); // y = Relu(x) + bias graph.remove_node(neg); graph.add_output(*relu.output(0), "rectified"); // Relu(x) is returned too graph.rename_input("x", "features"); graph.rename_output("y", "total"); const std::vector ins = graph.input_names(); const std::vector outs = graph.output_names(); std::printf("%s %s | %s %s\n", ins[0].c_str(), ins[1].c_str(), outs[0].c_str(), outs[1].c_str()); // features bias | total rectified graph.remove_output("rectified"); // the Relu that computed it stays: the Add reads it std::printf("%s | %s\n", graph.output_names().front().c_str(), producers(add).c_str()); // total | Relu bias return 0; } ``` ```python title="graph_io.py" (relu,) = graph.find_nodes(OpCode.Relu) (neg,) = graph.find_nodes(OpCode.Neg) (add,) = graph.find_nodes(OpCode.Add) bias = graph.add_input(crt.TensorSpec("bias", crt.float32, [2, 3])) # a new input, bound at run() graph.set_input(add, 1, bias.output(0)) # y = Relu(x) + bias graph.remove_node(neg) graph.add_output(relu.output(0), "rectified") # Relu(x) is returned too graph.rename_input("x", "features") graph.rename_output("y", "total") print(graph.input_names(), graph.output_names()) # ['features', 'bias'] ['total', 'rectified'] graph.remove_output("rectified") # the Relu that computed it stays: the Add reads it print(graph.output_names(), producers(add)) # ['total'] ['Relu', 'bias'] ``` Edit the inputs and outputs to expose an intermediate value, to feed a value from outside the graph, or to match the names a caller binds. ## Optimize, finalize and run the edited graph After the edits, `optimize()` runs the graph optimizer when you want it, and `finalize()` makes the graph runnable, after which it refuses edits. `run()` binds the inputs in `input_names()` order, the new input included. Each program checks the result against a reference it computes itself; the Python program also imports numpy as `np` for that. ```cpp title="after_the_edits.cpp" int main() { ModelGraph graph = trace_model(); const Node neg = graph.find_nodes(OpCode::Neg).front(); const Node add = graph.find_nodes(OpCode::Add).front(); const Node bias = graph.add_input({"bias", DataType::Float32, {2, 3}}); graph.set_input(add, 1, *bias.output(0)); // y = Relu(x) + bias graph.remove_node(neg); graph.optimize(); // optional: the graph optimizer runs over the edited graph graph.finalize(); // the graph runs from here on, and refuses edits const std::vector x = {1.0F, -2.0F, 3.0F, -4.0F, 5.0F, -6.0F}; const std::vector results = graph.run( {Tensor::from_data(x.data(), {2, 3}, DataType::Float32), Tensor::full({2, 3}, 0.5, DataType::Float32)}); const std::vector y = results.front().reshape({-1}).item_as_vec(); bool matches = y.size() == x.size(); for (std::size_t i = 0; matches && i < x.size(); ++i) { matches = y[i] == (x[i] > 0.0F ? x[i] : 0.0F) + 0.5F; // the reference, Relu(x) + 0.5, by hand } for (std::size_t i = 0; i < y.size(); ++i) std::printf("%s%g", i == 0 ? "" : " ", y[i]); std::printf("\n%s\n", matches ? "true" : "false"); // 1.5 0.5 3.5 0.5 5.5 0.5 // true return 0; } ``` ```python title="after_the_edits.py" (neg,) = graph.find_nodes(OpCode.Neg) (add,) = graph.find_nodes(OpCode.Add) bias = graph.add_input(crt.TensorSpec("bias", crt.float32, [2, 3])) graph.set_input(add, 1, bias.output(0)) # y = Relu(x) + bias graph.remove_node(neg) graph.optimize() # optional: the graph optimizer runs over the edited graph graph.finalize() # the graph runs from here on, and refuses edits x = np.array([[1.0, -2.0, 3.0], [-4.0, 5.0, -6.0]], np.float32) b = np.full((2, 3), 0.5, np.float32) (y,) = graph.run([crt.tensor(x), crt.tensor(b)]) print(y.numpy().tolist()) # [[1.5, 0.5, 3.5], [0.5, 5.5, 0.5]] print(np.array_equal(y.numpy(), np.maximum(x, 0) + b)) # True (numpy states the reference) ``` Finalize once the graph has every edit it needs, since a finalized graph takes none. ## Views across edits A `Node`, `Value` or `Edge` view taken before an edit finds its node, value or edge again by its key and keeps answering, and it refuses once an edit removed or renamed what it names, as [Views across edits](query-a-graph.mdx#views-across-edits) on the Query a graph page shows.