ClikaRT at a glance

ClikaRT is CLIKA's inference runtime, a C++ library (with Python and Kotlin bindings) that loads models and runs them on CPUs, GPUs, and other hardware accelerators through one public API. It is not a training framework, and not a bundle of separate CUDA, Vulkan and Metal wrappers. You link one library, include one header, and the same code runs on every backend the runtime ships.

The mental model

Five ideas carry the whole library, in the order you meet them when you build.

1. Everything arrives in one bundle. A directory with the public headers, the libraries per platform, a CMake package and the examples. find_package(ClikaRT CONFIG) and the target ClikaRT::ClikaRT are the entire integration.

2. Models load from files you already have. ClikaRT::io reads safetensors, GGUF and NumPy checkpoints, plus images and audio, and a model checkpoint arrives as a name -> tensor map on the device you name.

   C++

   const NamedTensors weights = io::load_safetensors("model.safetensors");
   const Tensor w = weights.get("layer.weight");

   Python

   weights = crt.io.load_safetensors("model.safetensors")
   w = weights["layer.weight"]

   Kotlin

   The Kotlin binding carries no checkpoint reader: a model reaches an app through the model library (AutoModel.fromPretrained), or as an ONNX model or a compiled graph; tensors and operators carry the Ops surface, one function per operator.

3. Tensor is the core building block. Each tensor is assigned to a device (the CPU by default; .to(device) moves it), and ops:: operators run where their inputs live. The CPU backend is always present; CUDA, Vulkan and Metal load at run time where the machine has them. Everything returns values directly and raises ClikaRT::Error on failure.

   C++

   const Tensor a = Tensor::ones({2, 3}, DataType::Float32);
   const Tensor b = ops::add(a, a);
   std::printf("%s\n", b.to_string().c_str());

   Python

   a = crt.ones(2, 3)
   b = a + a
   print(b)

   Kotlin

   val a = Tensors.ones(longArrayOf(2, 3))
   val b = a + a  // operator extensions return a fresh tensor; operands stay live
   println(b.summary())

4. Execution is asynchronous by nature. An ops:: call dispatches work and returns; reads wait for the result, so you never observe unfinished bytes.

5. Serving is built in. The serving runtime adds sessions, continuous batching and pipelines, so the model you loaded answers requests. Declare a schema, serve it with a lambda, drive it through an executor:

   C++

   rt::FunctionModel model{schema};
   model.on_run_once("run", [](rt::PhaseContext& ctx) {
       ctx.outputs->set("y", ctx.inputs->get("x") * 2.0);
   });
   
   rt::Executor exec = rt::Executor::create(model);
   const rt::Response resp = exec.await(exec.enqueue(std::move(req)));

   Python

   The Python lane serves a traced graph: crt.trace captures a forward pass as a ModelGraph, which runs once finalized, and crt.runtime carries the Executor and Pipeline over it. Part 5 walks it.

   Kotlin

   The Kotlin binding carries the serving runtime (FunctionModel, Executor, Pipeline) and the model library's server (Modelverse.serve), on a phone and on a desktop JVM alike.

The first program series turns these into working programs, explaining each where it first appears; the serving runtime has its own example project.

Platforms

Linux (x86_64, arm64), Android (arm64), Windows (x86_64, arm64) and macOS (Apple silicon), one distribution per platform and architecture, and a distribution works out of the box on every machine of its class: the linux-arm64 dist runs on a Jetson the same way it runs on an arm64 server. System requirements has the platform and accelerator tables.