--- title: Devices and the async model sidebar_label: 3. Devices and async description: "Discover the machine's devices, place tensors on them, and understand dispatch vs ready, and why host reads are always safe." --- The same program from parts 1-2 runs unchanged on a GPU. Data placement is the only new ingredient. This part adds device discovery and `.to(device)`, then explains the execution model behind every `ops::` call you have made so far. ## The program ```cpp title="main.cpp" #include #include using ClikaRT::DataType; using ClikaRT::Device; using ClikaRT::device::ComputeAPI; namespace device = ClikaRT::device; using ClikaRT::Tensor; namespace ops = ClikaRT::ops; // The best device this machine has, probed at run time. An unavailable // backend is a fact, not an error: the predicate answers false. namespace { Device pick_device() { if (device::is_cuda_available()) return Device::cuda(); if (device::is_vulkan_available()) return Device::vulkan(); if (device::is_metal_available()) return Device::metal(); return Device::cpu(); } } // namespace int main() { // What is this machine carrying? enumerate_devices lists the concrete // handles a backend exposes; get_device_properties describes one. for (ComputeAPI api : {ComputeAPI::CPU, ComputeAPI::CUDA, ComputeAPI::Vulkan, ComputeAPI::Metal}) { for (Device dev : device::enumerate_devices(api)) { const device::DeviceProperties p = device::get_device_properties(dev); std::printf("%-7s %d: %s\n", device::compute_api_name(api), dev.index, p.name.c_str()); } } const Device dev = pick_device(); std::printf("running on %s\n", device::compute_api_name(dev.api)); // .to(device) moves data; ops run where their inputs live. const Tensor a = Tensor::ones({512, 512}, DataType::Float32).to(dev); // This call DISPATCHES the matmul and returns. The kernel runs in its // own time; nothing here waits for it. const Tensor c = ops::matmul(a, a); // A host read is where the wait lands: it synchronizes first, so you // never observe unfinished bytes. Every element is 512 (= K). std::printf("every element = %.0f\n", ops::amax(c).item()); return 0; } ``` ```python title="main.py" import clika_runtime as crt def main() -> None: # The best device this machine has: Device.gpu() probes the available # accelerators and FALLS BACK to the CPU when none is present; it never # fails, so the same script runs everywhere. dev = crt.Device.gpu() print(f"running on {dev!r}") # Factories take the device directly; ops run where their inputs live. a = crt.ones(512, 512, device=dev) # This call DISPATCHES the matmul and returns. The kernel runs in its # own time; nothing here waits for it. c = a @ a # A host read is where the wait lands: it synchronizes first, so you # never observe unfinished bytes. Every element is 512 (= K). print(f"every element = {c.amax().item():.0f}") if __name__ == "__main__": main() ``` ```kotlin title="Main.kt" import io.clika.runtime.Backends import io.clika.runtime.ClikaRt import io.clika.runtime.ComputeApi import io.clika.runtime.Device import io.clika.runtime.Ops import io.clika.runtime.Tensors import java.util.Locale // The best device this machine has, probed at run time. A backend that is // not in this build, or whose devices do not serve, is a fact the probe // answers, never an error. private fun pickDevice(): Device { for (api in listOf(ComputeApi.CUDA, ComputeApi.VULKAN, ComputeApi.METAL)) { if (Backends.isAvailable(api) && Backends.deviceStatus(api) == null && Backends.deviceCount(api) > 0) { return Device(api, 0) } } return Device(ComputeApi.CPU) } fun main() { ClikaRt.load() // What is this machine carrying? Backends.devices lists the devices a // backend exposes; Device.properties() describes one. for (api in listOf(ComputeApi.CPU, ComputeApi.CUDA, ComputeApi.VULKAN, ComputeApi.METAL)) { if (!Backends.isAvailable(api)) continue for (dev in Backends.devices(api)) { println("%-7s %d: %s".format(Locale.ROOT, api.label, dev.index, dev.properties().name)) } } val dev = pickDevice() println("running on ${dev.api.label}") // Placement is a factory argument; operators run where their inputs live. val a = Tensors.ones(longArrayOf(512, 512), where = dev) // This call DISPATCHES the matmul and returns. The kernel runs in its // own time; nothing here waits for it. val c = Ops.matmul(a, a) // A host read is where the wait lands: item() synchronizes first, so // you never observe unfinished bytes. Every element is 512 (= K). val m = Ops.amax(c) println("every element = ${"%.0f".format(Locale.ROOT, m.item())}") listOf(a, c, m).forEach { it.release() } } ``` On a machine with an NVIDIA GPU: ```text CPU 0: AMD CUDA 0: NVIDIA RTX PRO 6000 Blackwell Workstation Edition CUDA 1: NVIDIA RTX PRO 6000 Blackwell Workstation Edition Vulkan 0: NVIDIA RTX PRO 6000 Blackwell Workstation Edition Vulkan 1: NVIDIA RTX PRO 6000 Blackwell Workstation Edition running on CUDA every element = 512 ``` The same binary on a CPU-only machine lists only the CPU and runs there; no rebuild, no configuration. ## Devices and backends A `Device` is a backend API plus a zero-based index. `Device::cuda(1)` is the second CUDA GPU, and `Device::cpu()` is the default everything starts on. Every distribution has the CPU backend compiled in; CUDA, Vulkan and Metal are shared libraries the runtime loads on demand, the first time something asks. `is_backend_available(api)` (and the shorthands `is_cuda_available()` and friends) answers whether that load works on this machine; `enumerate_devices(api)` returns the concrete handles, and an empty list is a valid answer, not an error. This is why the same binary runs everywhere. Absent hardware costs you a branch, not a build configuration. `.to(device)` returns the tensor moved (a no-op copy if it is already there), and operators run on the device their inputs live on; there is no global "current device" to set. ## Dispatch is not execution ClikaRT is asynchronous by nature. An `ops::` call **dispatches** work and returns; the kernel runs and the result becomes **ready** in its own time. On the default CPU stream ops happen to run inline, which is why parts 1-2 never confronted this. On an accelerator, or a worker stream made with `Stream::create`, the dispatch returns first and the compute overlaps with your code. `t.synchronize()` blocks until `t`'s pending work is done; `t.on_complete(callback)` is the push-style equivalent, firing when the result is ready. ## Reading across the async boundary Two rules cover every host read: 1. **Reads wait for you.** `item()`, `item_as_vec()` and `const_data_ptr()` synchronize before handing back bytes. A read issued right after dispatching heavy work blocks until the result is real. The wait moves into the read; it never disappears. You can never observe garbage through the public read surface. 2. **Pointers are device pointers.** `const_data_ptr()` addresses the buffer on the tensor's own device. On a CUDA tensor that is CUDA memory; call `t.to(Device::cpu())` first to read it on the host. (`item` / `item_as_vec` do the host transfer for you.) So the failure mode is never corruption, it is a surprise stall, a "cheap" read that waited for a matmul. When latency matters, choose where the wait lands: an explicit `synchronize()`, an `on_complete` callback, or a read whose cost you have accepted. The `async` [example project](/examples.md) measures all of this with timers. Next: [part 4](04-your-first-pipeline.mdx). It loads weights from disk, computes, and reads results back.