Skip to main content

Devices and the async model

The same program from parts 1-2 runs unchanged on a GPU. Data placement is the only new ingredient. This part adds device discovery and .to(device), then explains the execution model behind every ops:: call you have made so far.

The program​

main.cpp
#include <cstdio>

#include <ClikaRT/clika_rt.h>

using ClikaRT::DataType;
using ClikaRT::Device;
using ClikaRT::device::ComputeAPI;
namespace device = ClikaRT::device;
using ClikaRT::Tensor;
namespace ops = ClikaRT::ops;

// The best device this machine has, probed at run time. An unavailable
// backend is a fact, not an error: the predicate answers false.
namespace {

Device pick_device() {
    if (device::is_cuda_available())   return Device::cuda();
    if (device::is_vulkan_available()) return Device::vulkan();
    if (device::is_metal_available())  return Device::metal();
    return Device::cpu();
}

}  // namespace


int main() {
    // What is this machine carrying? enumerate_devices lists the concrete
    // handles a backend exposes; get_device_properties describes one.
    for (ComputeAPI api : {ComputeAPI::CPU, ComputeAPI::CUDA,
                           ComputeAPI::Vulkan, ComputeAPI::Metal}) {
        for (Device dev : device::enumerate_devices(api)) {
            const device::DeviceProperties p = device::get_device_properties(dev);
            std::printf("%-7s %d: %s\n",
                        device::compute_api_name(api), dev.index, p.name.c_str());
        }
    }

    const Device dev = pick_device();
    std::printf("running on %s\n", device::compute_api_name(dev.api));

    // .to(device) moves data; ops run where their inputs live.
    const Tensor a = Tensor::ones({512, 512}, DataType::Float32).to(dev);

    // This call DISPATCHES the matmul and returns. The kernel runs in its
    // own time; nothing here waits for it.
    const Tensor c = ops::matmul(a, a);

    // A host read is where the wait lands: it synchronizes first, so you
    // never observe unfinished bytes. Every element is 512 (= K).
    std::printf("every element = %.0f\n", ops::amax(c).item<float>());
    return 0;
}

On a machine with an NVIDIA GPU:

CPU     0: AMD
CUDA    0: NVIDIA RTX PRO 6000 Blackwell Workstation Edition
CUDA    1: NVIDIA RTX PRO 6000 Blackwell Workstation Edition
Vulkan  0: NVIDIA RTX PRO 6000 Blackwell Workstation Edition
Vulkan  1: NVIDIA RTX PRO 6000 Blackwell Workstation Edition
running on CUDA
every element = 512

The same binary on a CPU-only machine lists only the CPU and runs there; no rebuild, no configuration.

Devices and backends​

A Device is a backend API plus a zero-based index. Device::cuda(1) is the second CUDA GPU, and Device::cpu() is the default everything starts on. Every distribution has the CPU backend compiled in; CUDA, Vulkan and Metal are shared libraries the runtime loads on demand, the first time something asks. is_backend_available(api) (and the shorthands is_cuda_available() and friends) answers whether that load works on this machine; enumerate_devices(api) returns the concrete handles, and an empty list is a valid answer, not an error. This is why the same binary runs everywhere. Absent hardware costs you a branch, not a build configuration.

.to(device) returns the tensor moved (a no-op copy if it is already there), and operators run on the device their inputs live on; there is no global "current device" to set.

Dispatch is not execution​

ClikaRT is asynchronous by nature. An ops:: call dispatches work and returns; the kernel runs and the result becomes ready in its own time. On the default CPU stream ops happen to run inline, which is why parts 1-2 never confronted this. On an accelerator, or a worker stream made with Stream::create, the dispatch returns first and the compute overlaps with your code. t.synchronize() blocks until t's pending work is done; t.on_complete(callback) is the push-style equivalent, firing when the result is ready.

Reading across the async boundary​

Two rules cover every host read:

  1. Reads wait for you. item<T>(), item_as_vec<T>() and const_data_ptr() synchronize before handing back bytes. A read issued right after dispatching heavy work blocks until the result is real. The wait moves into the read; it never disappears. You can never observe garbage through the public read surface.
  2. Pointers are device pointers. const_data_ptr() addresses the buffer on the tensor's own device. On a CUDA tensor that is CUDA memory; call t.to(Device::cpu()) first to read it on the host. (item / item_as_vec do the host transfer for you.)

So the failure mode is never corruption, it is a surprise stall, a "cheap" read that waited for a matmul. When latency matters, choose where the wait lands: an explicit synchronize(), an on_complete callback, or a read whose cost you have accepted. The async example project measures all of this with timers.

Next: part 4. It loads weights from disk, computes, and reads results back.