Devices and the async model

The same program from parts 1-2 runs unchanged on a GPU. Data placement is the only new ingredient. This part adds device discovery and .to(device), then explains the execution model behind every ops:: call you have made so far.

The program

C++

main.cpp

#include <cstdio>

#include <ClikaRT/clika_rt.h>

using ClikaRT::DataType;
using ClikaRT::Device;
using ClikaRT::device::ComputeAPI;
namespace device = ClikaRT::device;
using ClikaRT::Tensor;
namespace ops = ClikaRT::ops;

// The best device this machine has, probed at run time. An unavailable
// backend is a fact, not an error: the predicate answers false.
namespace {

Device pick_device() {
    if (device::is_cuda_available())   return Device::cuda();
    if (device::is_vulkan_available()) return Device::vulkan();
    if (device::is_metal_available())  return Device::metal();
    return Device::cpu();
}

}  // namespace


int main() {
    // What is this machine carrying? enumerate_devices lists the concrete
    // handles a backend exposes; get_device_properties describes one.
    for (ComputeAPI api : {ComputeAPI::CPU, ComputeAPI::CUDA,
                           ComputeAPI::Vulkan, ComputeAPI::Metal}) {
        for (Device dev : device::enumerate_devices(api)) {
            const device::DeviceProperties p = device::get_device_properties(dev);
            std::printf("%-7s %d: %s\n",
                        device::compute_api_name(api), dev.index, p.name.c_str());
        }
    }

    const Device dev = pick_device();
    std::printf("running on %s\n", device::compute_api_name(dev.api));

    // .to(device) moves data; ops run where their inputs live.
    const Tensor a = Tensor::ones({512, 512}, DataType::Float32).to(dev);

    // This call DISPATCHES the matmul and returns. The kernel runs in its
    // own time; nothing here waits for it.
    const Tensor c = ops::matmul(a, a);

    // A host read is where the wait lands: it synchronizes first, so you
    // never observe unfinished bytes. Every element is 512 (= K).
    std::printf("every element = %.0f\n", ops::amax(c).item<float>());
    return 0;
}

Python

main.py

import clika_runtime as crt


def main() -> None:
    # The best device this machine has: Device.gpu() probes the available
    # accelerators and FALLS BACK to the CPU when none is present; it never
    # fails, so the same script runs everywhere.
    dev = crt.Device.gpu()
    print(f"running on {dev!r}")

    # Factories take the device directly; ops run where their inputs live.
    a = crt.ones(512, 512, device=dev)

    # This call DISPATCHES the matmul and returns. The kernel runs in its
    # own time; nothing here waits for it.
    c = a @ a

    # A host read is where the wait lands: it synchronizes first, so you
    # never observe unfinished bytes. Every element is 512 (= K).
    print(f"every element = {c.amax().item():.0f}")


if __name__ == "__main__":
    main()

Kotlin

Main.kt

import io.clika.runtime.Backends
import io.clika.runtime.ClikaRt
import io.clika.runtime.ComputeApi
import io.clika.runtime.Device
import io.clika.runtime.Ops
import io.clika.runtime.Tensors
import java.util.Locale

// The best device this machine has, probed at run time. A backend that is
// not in this build, or whose devices do not serve, is a fact the probe
// answers, never an error.
private fun pickDevice(): Device {
    for (api in listOf(ComputeApi.CUDA, ComputeApi.VULKAN, ComputeApi.METAL)) {
        if (Backends.isAvailable(api) && Backends.deviceStatus(api) == null && Backends.deviceCount(api) > 0) {
            return Device(api, 0)
        }
    }
    return Device(ComputeApi.CPU)
}

fun main() {
    ClikaRt.load()

    // What is this machine carrying? Backends.devices lists the devices a
    // backend exposes; Device.properties() describes one.
    for (api in listOf(ComputeApi.CPU, ComputeApi.CUDA, ComputeApi.VULKAN, ComputeApi.METAL)) {
        if (!Backends.isAvailable(api)) continue
        for (dev in Backends.devices(api)) {
            println("%-7s %d: %s".format(Locale.ROOT, api.label, dev.index, dev.properties().name))
        }
    }

    val dev = pickDevice()
    println("running on ${dev.api.label}")

    // Placement is a factory argument; operators run where their inputs live.
    val a = Tensors.ones(longArrayOf(512, 512), where = dev)

    // This call DISPATCHES the matmul and returns. The kernel runs in its
    // own time; nothing here waits for it.
    val c = Ops.matmul(a, a)

    // A host read is where the wait lands: item() synchronizes first, so
    // you never observe unfinished bytes. Every element is 512 (= K).
    val m = Ops.amax(c)
    println("every element = ${"%.0f".format(Locale.ROOT, m.item())}")

    listOf(a, c, m).forEach { it.release() }
}

On a machine with an NVIDIA GPU:

CPU     0: AMD
CUDA    0: NVIDIA RTX PRO 6000 Blackwell Workstation Edition
CUDA    1: NVIDIA RTX PRO 6000 Blackwell Workstation Edition
Vulkan  0: NVIDIA RTX PRO 6000 Blackwell Workstation Edition
Vulkan  1: NVIDIA RTX PRO 6000 Blackwell Workstation Edition
running on CUDA
every element = 512

The same binary on a CPU-only machine lists only the CPU and runs there; no rebuild, no configuration.

Devices and backends

A Device is a backend API plus a zero-based index. Device::cuda(1) is the second CUDA GPU, and Device::cpu() is the default everything starts on. Every distribution has the CPU backend compiled in; CUDA, Vulkan and Metal are shared libraries the runtime loads on demand, the first time something asks. is_backend_available(api) (and the shorthands is_cuda_available() and friends) answers whether that load works on this machine; enumerate_devices(api) returns the concrete handles, and an empty list is a valid answer, not an error. This is why the same binary runs everywhere. Absent hardware costs you a branch, not a build configuration.

.to(device) returns the tensor moved (a no-op copy if it is already there), and operators run on the device their inputs live on; there is no global "current device" to set.

Dispatch is not execution

ClikaRT is asynchronous by nature. An ops:: call dispatches work and returns; the kernel runs and the result becomes ready in its own time. On the default CPU stream ops happen to run inline, which is why parts 1-2 never confronted this. On an accelerator, or a worker stream made with Stream::create, the dispatch returns first and the compute overlaps with your code. t.synchronize() blocks until t's pending work is done; t.on_complete(callback) is the push-style equivalent, firing when the result is ready.

Reading across the async boundary

Two rules cover every host read:

1. Reads wait for you. item<T>(), item_as_vec<T>() and const_data_ptr() synchronize before handing back bytes. A read issued right after dispatching heavy work blocks until the result is real. The wait moves into the read; it never disappears. You can never observe garbage through the public read surface.
2. Pointers are device pointers. const_data_ptr() addresses the buffer on the tensor's own device. On a CUDA tensor that is CUDA memory; call t.to(Device::cpu()) first to read it on the host. (item / item_as_vec do the host transfer for you.)

So the failure mode is never corruption, it is a surprise stall, a "cheap" read that waited for a matmul. When latency matters, choose where the wait lands: an explicit synchronize(), an on_complete callback, or a read whose cost you have accepted. The async example project measures all of this with timers.

Next: part 4. It loads weights from disk, computes, and reads results back.