Skip to main content

Edit a graph

A ModelGraph that you trace or compile can be changed before you call finalize(). You can move the reads of a value to another value, remove or bypass a node, change what a node reads, rename a node, and add, remove or rename the graph's inputs and outputs. Each edit checks everything before it changes anything, so a refused edit leaves the graph as it was and says what it refused. The edits are available from C++ and Python, under the same names.

A graph to edit​

A trace returns the graph as built, with every operator as written and nothing optimized or finalized, so it is ready to edit. The model below computes y = Relu(x) + Neg(x): x feeds a Relu and a Neg, and one Add reads both. Every example on this page uses it. label names a node by its input name or its operator, labels lists a graph's nodes, and producers lists the nodes a node reads, one per input port.

Edits run before finalize(). A finalized graph serves run() and refuses every edit with the code name FAILED_PRECONDITION; to edit it, trace or compile the model again.

get_a_graph.cpp
#include <cstdio>
#include <string>
#include <vector>

#include <ClikaRT/clika_rt.h>

using ClikaRT::DataType;
using ClikaRT::Error;
using ClikaRT::Tensor;
using ClikaRT::graph::ModelGraph;
using ClikaRT::graph::Node;
using ClikaRT::graph::NodeKind;
using ClikaRT::graph::OpCode;
using ClikaRT::graph::Value;
namespace ops = ClikaRT::ops;

namespace {

// y = Relu(x) + Neg(x): x feeds a Relu and a Neg, both read by one Add.
std::vector<Tensor> model(const std::vector<Tensor>& inputs) {
    const Tensor rectified = ops::relu(inputs[0]);
    const Tensor negated = ops::neg(inputs[0]);
    return {ops::add(rectified, negated)};
}

// A node's label: an input's name, an operator's code.
std::string label(const Node& node) {
    return node.kind() == NodeKind::Input ? node.name() : std::string(ClikaRT::graph::op_code_name(node.op_code()));
}

// The labels of `nodes`, separated by spaces.
std::string labels(const std::vector<Node>& nodes) {
    std::string out;
    for (const Node& node : nodes) out += (out.empty() ? "" : " ") + label(node);
    return out;
}

// The labels of the nodes that produce what `node` reads, one per input port.
std::string producers(const Node& node) {
    std::string out;
    for (const Value& value : node.inputs()) out += (out.empty() ? "" : " ") + label(*value.producer());
    return out;
}

// A trace returns the graph as built: every operator as written, nothing optimized or finalized.
ModelGraph trace_model() {
    const std::vector<ClikaRT::spec::TensorSpec> signature = {{"x", DataType::Float32, {2, 3}}};
    const std::vector<std::string> outputs = {"y"};
    return ClikaRT::graph::trace(model, signature, "edit", outputs);
}

}  // namespace

int main() {
    const ModelGraph graph = trace_model();
    const Node add = graph.find_nodes(OpCode::Add).front();
    std::printf("%s | %s\n", labels(graph.nodes()).c_str(), producers(add).c_str());   // x Relu Neg Add | Relu Neg
    std::printf("%s %s\n", graph.output_names().front().c_str(), graph.is_finalized() ? "true" : "false");   // y false

    // Edits run before finalize(): a finalized graph refuses every one of them.
    ModelGraph done = trace_model();
    done.finalize();
    try {
        done.rename_output("y", "total");
    } catch (const Error& error) {
        std::printf("%s\n", error.code_name().c_str());   // FAILED_PRECONDITION
    }
    return 0;
}

Edit a graph when a model needs a change its source does not make: an operator to remove before deployment, a value to expose or feed, or a name a caller binds.

Each program on this page is complete and runs on its own. From here on, a block shows the part of its program that follows the opening lines the first block shows (the includes or imports, model, label, labels, producers and the trace).

A refused edit changes nothing​

Every edit checks its arguments and the graph's rules before it changes anything, so a refused edit leaves the nodes, the inputs, the outputs and the constants as they were. Its message names the call and what it refused. The code name is INVALID_ARGUMENT for an argument the edit cannot take, and FAILED_PRECONDITION for a finalized graph. In C++ the edit throws a ClikaRT::Error (a Result carries it in a build without exceptions); in Python it raises InvalidArgumentError, whose code_name says which. Here rename_node refuses an input and names the call that renames one.

refused_edit.cpp
int main() {
    ModelGraph graph = trace_model();
    const std::string before = labels(graph.nodes());
    try {
        graph.rename_node(*graph.node("x"), "features");   // an input is renamed with rename_input()
    } catch (const Error& error) {
        std::printf("%s | %s\n", error.code_name().c_str(), error.what());
        // INVALID_ARGUMENT | rename_node: 'x' is a graph input; rename it with rename_input()
    }
    std::printf("%s %s\n", labels(graph.nodes()) == before ? "true" : "false",
                graph.input_names().front().c_str());   // true x
    return 0;
}

Branch on the code name and report the message; Handle errors by code covers the channels every failure carries.

Rewire the reads of a value​

replace_all_uses_with(old_value, replacement) moves every read of a value to another one: each operator input it feeds, and each graph output it returns, which keeps its name. remove_node(node) removes an operator that nothing reads, and refuses one that is still read, naming the reader to rewire or remove first. bypass_node(node) hands a node's readers the value on its input (the first input and output, unless you name the ports) and removes the node. A value takes another's place only with the same dtype and dims.

rewire.cpp
int main() {
    ModelGraph graph = trace_model();
    const Node relu = graph.find_nodes(OpCode::Relu).front();
    const Node neg = graph.find_nodes(OpCode::Neg).front();
    const Node add = graph.find_nodes(OpCode::Add).front();
    graph.replace_all_uses_with(*neg.output(0), *graph.node("x")->output(0));   // the Add reads x where it read Neg(x)
    graph.remove_node(neg);                                                      // nothing reads the Neg now
    std::printf("%s\n", labels(graph.nodes()).c_str());                          // x Relu Add
    graph.bypass_node(relu);   // the Add reads the Relu's input, x, and the Relu goes
    std::printf("%s | %s\n", labels(graph.nodes()).c_str(), producers(add).c_str());   // x Add | x x
    return 0;
}

Rewire a graph to remove an operator a deployment does not need, or to let readers take a value the graph already computes.

Change what a node reads​

set_input(node, port, value) gives one input port another value; the port must read a value already. set_constant_input(node, port, tensor) binds a new constant holding the tensor's bytes to the port: the graph keeps a handle to the bytes, with no copy, and places them with the graph at finalize(). add_constant(tensor) makes a constant that nothing reads yet, for set_input or replace_all_uses_with to wire in, and finalize() drops a constant that nothing reads by then. A weight an operator holds itself, such as a compiled MatMul's, stays as it is, and an edit that would replace it refuses.

node_inputs.cpp
int main() {
    ModelGraph graph = trace_model();
    const Node relu = graph.find_nodes(OpCode::Relu).front();
    const Node neg = graph.find_nodes(OpCode::Neg).front();
    const Node add = graph.find_nodes(OpCode::Add).front();
    graph.set_input(add, 1, *relu.output(0));      // port 1 reads Relu(x) in place of Neg(x)
    graph.remove_node(neg);                        // which nothing reads now
    std::printf("%s\n", producers(add).c_str());   // Relu Relu

    // Port 1 reads a constant holding the tensor's bytes.
    graph.set_constant_input(add, 1, Tensor::full({2, 3}, 1.0, DataType::Float32));
    std::printf("%s %zu\n", add.input(1)->is_constant() ? "true" : "false", graph.constants().size());   // true 1

    const Value half = graph.add_constant(Tensor::full({2, 3}, 0.5, DataType::Float32));   // nothing reads it yet
    graph.set_input(add, 1, half);   // port 1 reads it, and the ones, read by nothing, go
    std::printf("%s %zu\n", *add.input(1) == half ? "true" : "false", graph.constants().size());   // true 1
    return 0;
}

Change a node's inputs to feed a constant of your own into the graph, or to point an operator at another producer.

Rename a node​

rename_node(node, name) renames an operator, and its outputs keep their names; an input is renamed with rename_input instead. A view of the old name is gone afterwards, so look the node up again with node(name).

rename.cpp
int main() {
    ModelGraph graph = trace_model();
    graph.rename_node(graph.find_nodes(OpCode::Add).front(), "total");   // views of the old name are gone afterwards
    const Node total = *graph.node("total");                              // so find the node again by its new one
    std::printf("%s | %s | %s\n", label(total).c_str(), producers(total).c_str(),
                graph.output_names().front().c_str());   // Add | Relu Neg | y
    return 0;
}

Rename nodes to give them the names your own tools and reports use.

Edit the graph's inputs and outputs​

add_input(spec) adds a graph input with the spec's name, dtype and dims (a dynamic dim takes its size at run()) and returns its input node, whose value the other edits take. add_output(value, name) returns a value under a new output name, and remove_output(name) stops returning one, while the node that computes it stays. rename_input(name, new_name) and rename_output(name, new_name) rename an input and an output. Input and output names stay unique, and a name a KV cache layer binds stays as it is.

graph_io.cpp
int main() {
    ModelGraph graph = trace_model();
    const Node relu = graph.find_nodes(OpCode::Relu).front();
    const Node neg = graph.find_nodes(OpCode::Neg).front();
    const Node add = graph.find_nodes(OpCode::Add).front();
    const Node bias = graph.add_input({"bias", DataType::Float32, {2, 3}});   // a new input, bound at run()
    graph.set_input(add, 1, *bias.output(0));                                // y = Relu(x) + bias
    graph.remove_node(neg);
    graph.add_output(*relu.output(0), "rectified");   // Relu(x) is returned too
    graph.rename_input("x", "features");
    graph.rename_output("y", "total");
    const std::vector<std::string> ins = graph.input_names();
    const std::vector<std::string> outs = graph.output_names();
    std::printf("%s %s | %s %s\n", ins[0].c_str(), ins[1].c_str(), outs[0].c_str(), outs[1].c_str());
    // features bias | total rectified
    graph.remove_output("rectified");   // the Relu that computed it stays: the Add reads it
    std::printf("%s | %s\n", graph.output_names().front().c_str(), producers(add).c_str());   // total | Relu bias
    return 0;
}

Edit the inputs and outputs to expose an intermediate value, to feed a value from outside the graph, or to match the names a caller binds.

Optimize, finalize and run the edited graph​

After the edits, optimize() runs the graph optimizer when you want it, and finalize() makes the graph runnable, after which it refuses edits. run() binds the inputs in input_names() order, the new input included. Each program checks the result against a reference it computes itself; the Python program also imports numpy as np for that.

after_the_edits.cpp
int main() {
    ModelGraph graph = trace_model();
    const Node neg = graph.find_nodes(OpCode::Neg).front();
    const Node add = graph.find_nodes(OpCode::Add).front();
    const Node bias = graph.add_input({"bias", DataType::Float32, {2, 3}});
    graph.set_input(add, 1, *bias.output(0));   // y = Relu(x) + bias
    graph.remove_node(neg);

    graph.optimize();   // optional: the graph optimizer runs over the edited graph
    graph.finalize();   // the graph runs from here on, and refuses edits
    const std::vector<float> x = {1.0F, -2.0F, 3.0F, -4.0F, 5.0F, -6.0F};
    const std::vector<Tensor> results = graph.run(
        {Tensor::from_data(x.data(), {2, 3}, DataType::Float32), Tensor::full({2, 3}, 0.5, DataType::Float32)});
    const std::vector<float> y = results.front().reshape({-1}).item_as_vec<float>();
    bool matches = y.size() == x.size();
    for (std::size_t i = 0; matches && i < x.size(); ++i) {
        matches = y[i] == (x[i] > 0.0F ? x[i] : 0.0F) + 0.5F;   // the reference, Relu(x) + 0.5, by hand
    }
    for (std::size_t i = 0; i < y.size(); ++i) std::printf("%s%g", i == 0 ? "" : " ", y[i]);
    std::printf("\n%s\n", matches ? "true" : "false");
    // 1.5 0.5 3.5 0.5 5.5 0.5
    // true
    return 0;
}

Finalize once the graph has every edit it needs, since a finalized graph takes none.

Views across edits​

A Node, Value or Edge view taken before an edit finds its node, value or edge again by its key and keeps answering, and it refuses once an edit removed or renamed what it names, as Views across edits on the Query a graph page shows.