--- title: Wrap existing memory without copying description: "Tensor::from_blob over buffers your application already owns: the borrow and adopt contracts, strided views, and what happens on a device move." --- Your data already sits in memory that some other part of the program owns: a decoder's output buffer, an arena, a mapped file, another library's array. `Tensor::from_blob` wraps that memory as a tensor with no copy; the tensor's data pointer is your pointer. What needs deciding is ownership, and the API makes the two contracts explicit: - **No deleter passed: borrowed.** You keep ownership. The buffer must stay alive, and its layout unchanged, for as long as the tensor or any view of it is in use. Writes through the buffer are visible through the tensor and the other way around; it is the same memory. - **Deleter passed: adopted.** The tensor takes ownership and calls `deleter(data)` once, when the last reference drops. The Python variant borrows a numpy array's memory. ## Borrow a buffer and compute on it ```cpp title="borrow.cpp" #include #include #include using ClikaRT::DataType; using ClikaRT::Tensor; namespace ops = ClikaRT::ops; int main() { std::vector buf(8, 1.0F); // memory the application owns const Tensor view = Tensor::from_blob(buf.data(), {8}, DataType::Float32); std::printf("shares memory: %s\n", view.const_data_ptr() == buf.data() ? "yes (no copy)" : "no"); std::printf("sum = %g\n", ops::sum(view).item()); buf[0] = 100.0F; // write through the buffer... std::printf("sum after buf[0] = 100: %g\n", ops::sum(view).item()); return 0; } ``` ```python title="main.py" import numpy as np import clika_runtime as crt def main() -> None: # Borrow: from_numpy wraps the array's own memory, no copy. The array # and the tensor see the same bytes, mutation aliases both ways, and # the tensor keeps the array alive. The array must be writable and # contiguous; the refusals name the fix. buf = np.zeros(4, dtype=np.float32) view = crt.from_numpy(buf) buf[0] = 100.0 print(f"sum after buf[0] = 100: {view.sum().item():g}") # The other direction: numpy() on a CPU tensor is a zero-copy view of # the tensor's memory; a device tensor asks you to move it first # (t.to('cpu').numpy()). t = crt.ones(2, 2) arr = t.numpy() print(f"shared bytes: {arr.sum():g}") # A non-contiguous array does not borrow; the error names the fix. try: crt.from_numpy(np.zeros((4, 4), dtype=np.float32)[:, ::2]) except TypeError as e: print(f"non-contiguous refused: {e}") if __name__ == "__main__": main() ``` `crt.tensor(array)` stays the copying entry when an independent tensor is wanted. ```text shares memory: yes (no copy) sum = 8 sum after buf[0] = 100: 107 ``` The borrow contract in one sentence: the runtime never reuses or overwrites borrowed memory, and in exchange you guarantee it outlives every tensor that sees it. A `vector` that reallocates (or a stack buffer that goes out of scope) under a live view is the bug this contract exists to name. ## Hand ownership over with a deleter When the producer wants to fire and forget, pass a deleter. The tensor (and every tensor computed from it) keeps the buffer alive; the deleter runs exactly once, when the last reference drops, and it runs in your runtime: an exception it throws never crosses the library boundary. It also frees the runtime to reuse the buffer as scratch, which the borrow contract forbids. ```cpp title="adopt.cpp" #include #include #include #include using ClikaRT::DataType; using ClikaRT::Tensor; namespace ops = ClikaRT::ops; namespace { constexpr std::int64_t kCount = 8; // elements in the caller-owned buffer } // namespace int main() { float* buf = static_cast(std::malloc(kCount * sizeof(float))); for (std::int64_t i = 0; i < kCount; ++i) buf[i] = static_cast(i); { const Tensor adopted = Tensor::from_blob( buf, {kCount}, DataType::Float32, {}, [](void* p) { std::printf("deleter: buffer released\n"); std::free(p); }); std::printf("mean = %g\n", ops::mean(adopted).item()); std::printf("leaving the tensor's scope...\n"); } std::printf("scope closed\n"); return 0; } ``` Handing ownership over with a deleter is part of the C++ API today; the C++ tab shows it. The python borrow keeps the ARRAY as the owner: `crt.from_numpy(array)` holds the array alive for the tensor's lifetime, so no deleter changes hands. ```text mean = 3.5 leaving the tensor's scope... deleter: buffer released scope closed ``` The deleter must not throw (a throw is swallowed). Adoption is the right contract at module boundaries: the producer allocates, the consumer wraps and forgets the allocation ever existed. ## Wrap non-contiguous memory with strides The strided overload views memory that is not laid out contiguously, without rearranging a byte. Strides are in elements, one per dimension. A worked case: cropping a region of interest out of a pitched image buffer, the layout every camera API and GPU readback hands you (rows padded to a pitch wider than the image). ```cpp title="strided_roi.cpp" #include #include #include #include using ClikaRT::DataType; using ClikaRT::Tensor; namespace ops = ClikaRT::ops; namespace { constexpr std::int64_t kRows = 4, kPitch = 6; // the whole image, row-major constexpr std::int64_t kRoiRow = 1, kRoiCol = 2; // where the region starts constexpr std::int64_t kRoiRows = 2, kRoiCols = 3; // its extent } // namespace int main() { // A 4x6 single-channel image, row-major. buf[r][c] = r*10 + c. std::vector buf(kRows * kPitch); for (std::int64_t r = 0; r < kRows; ++r) for (std::int64_t c = 0; c < kPitch; ++c) buf[r * kPitch + c] = static_cast(r * 10 + c); // The 2x3 region starting at row 1, column 2: shape {2, 3}, and the // ORIGINAL row pitch as the row stride. No pixel is copied. const Tensor roi = Tensor::from_blob(buf.data() + kRoiRow * kPitch + kRoiCol, {kRoiRows, kRoiCols}, {kPitch, 1}, DataType::Float32); std::printf("roi = %s\n", roi.to_string().c_str()); std::printf("sum = %g (12+13+14+22+23+24 = 108)\n", ops::sum(roi).item()); return 0; } ``` Wrapping strided memory with explicit strides is part of the C++ API today; the C++ tab shows it. `crt.from_numpy` takes a C-contiguous array; a strided source enters through `np.ascontiguousarray` (the refusal names it). ```text roi = Tensor(shape=[2, 3], dtype=Float32, device=CPU, numel=6, data=[12, 13, 14, 22, 23, 24]) sum = 108 (12+13+14+22+23+24 = 108) ``` The same shape covers any pitched or tiled layout: a submatrix of a row-major matrix, a plane in a planar image, a batch entry inside a larger allocation. `ops::contiguous` materializes an owned compact copy when a consumer needs one. ## Device moves and pinned memory `.to(device)` on a wrapped tensor behaves like on any other: on a discrete accelerator the move is a real transfer to device memory (the wrap saved the host-side copy, not the transfer), while unified-memory hardware moves for free. Two related notes. `from_data` is the copying cousin: it copies your bytes into an owned tensor so the source's lifetime stops mattering; take it when the buffer is short-lived and the tensor is not. And `from_blob`'s `pinned_for` parameter is tag-only: it asserts pages you already page-locked for a device, letting transfers take the pinned path; it cannot pin memory for you. ## Give the pool's idle reserve back The runtime side of the memory story: the pool keeps memory it handed out and got back (`MemoryStats::cached_bytes`), so the next allocation is cheap. After a model unloads, or before a second model must fit beside the first, that idle reserve is memory the device (on a shared-memory part, the host) cannot use for anything else. `device::release_cached_memory(device)` returns it to the driver or the OS: deferred reservations drain, parked buffers retire and empty blocks release, the same three steps the runtime takes before it reports out-of-memory. Memory still in use, or whose last use has not completed on the device, is never touched; a later call can release more once that work retires. Automatic placement calls it on a failed accelerator before the CPU fallback loads. ```cpp title="release_cached.cpp" #include #include using ClikaRT::DataType; using ClikaRT::Device; using ClikaRT::Tensor; static void report(const char* when) { const ClikaRT::device::MemoryStats s = ClikaRT::device::memory_stats(Device::cpu()); std::printf("%-14s active %8.1f MiB, cached %8.1f MiB\n", when, s.active_bytes / 1048576.0, s.cached_bytes / 1048576.0); } int main() { { // 256 MiB of Float32 work: the pool reserves real memory for it. Tensor big = Tensor::zeros({64, 1024, 1024}, DataType::Float32); big.synchronize(); report("in use:"); } // The tensor is gone, but the pool keeps its bytes idle for reuse. report("dropped:"); // Hand the idle reserve back to the system. Memory still in use, or // whose last use has not completed, is never touched. ClikaRT::device::release_cached_memory(Device::cpu()); report("released:"); return 0; } ``` ```text in use: active 256.0 MiB, cached 0.0 MiB dropped: active 0.0 MiB, cached 256.0 MiB released: active 0.0 MiB, cached 0.0 MiB ``` The bundle's `compute` example covers device movement and this wrap in its `03_data_movement` and `06_zero_copy` chapters; [the custom-operator guide](write-a-custom-operator.mdx) uses `from_blob` to hand a hand-written kernel's output back to the runtime.