Skip to main content

Wrap existing memory without copying

Your data already sits in memory that some other part of the program owns: a decoder's output buffer, an arena, a mapped file, another library's array. Tensor::from_blob wraps that memory as a tensor with no copy; the tensor's data pointer is your pointer. What needs deciding is ownership, and the API makes the two contracts explicit:

  • No deleter passed: borrowed. You keep ownership. The buffer must stay alive, and its layout unchanged, for as long as the tensor or any view of it is in use. Writes through the buffer are visible through the tensor and the other way around; it is the same memory.
  • Deleter passed: adopted. The tensor takes ownership and calls deleter(data) once, when the last reference drops.

The C samples abbreviate the api-table bootstrap that tutorial part 1 shows in full. The Python variant borrows a numpy array's memory; the Kotlin arm borrows a direct java.nio.ByteBuffer, the one JVM buffer with a stable native address (a heap buffer is refused with a readable error).

Borrow a buffer and compute on it​

borrow.cpp
#include <cstdio>
#include <vector>

#include "ClikaRT/clika_rt.h"

using ClikaRT::DataType;
using ClikaRT::Tensor;
namespace ops = ClikaRT::ops;

int main() {
    std::vector<float> buf(8, 1.0F);  // memory the application owns

    const Tensor view = Tensor::from_blob(buf.data(), {8}, DataType::Float32);
    std::printf("shares memory: %s\n",
                view.const_data_ptr() == buf.data() ? "yes (no copy)" : "no");
    std::printf("sum = %g\n", ops::sum(view).item<float>());

    buf[0] = 100.0F;  // write through the buffer...
    std::printf("sum after buf[0] = 100: %g\n", ops::sum(view).item<float>());
    return 0;
}
shares memory: yes (no copy)
sum = 8
sum after buf[0] = 100: 107

The borrow contract in one sentence: the runtime never reuses or overwrites borrowed memory, and in exchange you guarantee it outlives every tensor that sees it. A vector that reallocates (or a stack buffer that goes out of scope) under a live view is the bug this contract exists to name.

Hand ownership over with a deleter​

When the producer wants to fire and forget, pass a deleter. The tensor (and every tensor computed from it) keeps the buffer alive; the deleter runs exactly once, when the last reference drops, and it runs in your runtime: an exception it throws never crosses the library boundary. It also frees the runtime to reuse the buffer as scratch, which the borrow contract forbids.

adopt.cpp
#include <cstdio>
#include <cstdlib>

#include "ClikaRT/clika_rt.h"

using ClikaRT::DataType;
using ClikaRT::Tensor;
namespace ops = ClikaRT::ops;

int main() {
    float* buf = static_cast<float*>(std::malloc(8 * sizeof(float)));
    for (int i = 0; i < 8; ++i) buf[i] = static_cast<float>(i);

    {
        const Tensor adopted = Tensor::from_blob(
            buf, {8}, DataType::Float32, {},
            [](void* p) { std::printf("deleter: buffer released\n"); std::free(p); });
        std::printf("mean = %g\n", ops::mean(adopted).item<float>());
        std::printf("leaving the tensor's scope...\n");
    }
    std::printf("scope closed\n");
    return 0;
}
mean = 3.5
leaving the tensor's scope...
deleter: buffer released
scope closed

The deleter must not throw (a throw is swallowed). Adoption is the right contract at module boundaries: the producer allocates, the consumer wraps and forgets the allocation ever existed.

Wrap non-contiguous memory with strides​

The strided overload views memory that is not laid out contiguously, without rearranging a byte. Strides are in elements, one per dimension. A worked case: cropping a region of interest out of a pitched image buffer, the layout every camera API and GPU readback hands you (rows padded to a pitch wider than the image).

strided_roi.cpp
#include <cstdint>
#include <cstdio>
#include <vector>

#include "ClikaRT/clika_rt.h"

using ClikaRT::DataType;
using ClikaRT::Tensor;
namespace ops = ClikaRT::ops;

int main() {
    // A 4x6 single-channel image, row-major. buf[r][c] = r*10 + c.
    constexpr std::int64_t kPitch = 6;
    std::vector<float> buf(4 * kPitch);
    for (std::int64_t r = 0; r < 4; ++r)
        for (std::int64_t c = 0; c < kPitch; ++c) buf[r * kPitch + c] = (float)(r * 10 + c);

    // The 2x3 region starting at row 1, column 2: shape {2, 3}, and the
    // ORIGINAL row pitch as the row stride. No pixel is copied.
    const Tensor roi = Tensor::from_blob(buf.data() + 1 * kPitch + 2,
                                         {2, 3}, {kPitch, 1}, DataType::Float32);

    std::printf("roi = %s\n", roi.to_string().c_str());
    std::printf("sum = %g (12+13+14+22+23+24 = 108)\n", ops::sum(roi).item<float>());
    return 0;
}
roi = Tensor(shape=[2, 3], dtype=Float32, device=CPU, numel=6, data=[12, 13, 14, 22, 23, 24])
sum = 108 (12+13+14+22+23+24 = 108)

The same shape covers any pitched or tiled layout: a submatrix of a row-major matrix, a plane in a planar image, a batch entry inside a larger allocation. ops::contiguous materializes an owned compact copy when a consumer needs one.

Device moves and pinned memory​

.to(device) on a wrapped tensor behaves like on any other: on a discrete accelerator the move is a real transfer to device memory (the wrap saved the host-side copy, not the transfer), while unified-memory hardware moves for free. Two related notes. from_data is the copying cousin: it copies your bytes into an owned tensor so the source's lifetime stops mattering; take it when the buffer is short-lived and the tensor is not. And from_blob's pinned_for parameter is tag-only: it asserts pages you already page-locked for a device, letting transfers take the pinned path; it cannot pin memory for you.

Give the pool's idle reserve back​

The runtime side of the memory story: the pool keeps memory it handed out and got back (MemoryStats::cached_bytes), so the next allocation is cheap. After a model unloads, or before a second model must fit beside the first, that idle reserve is memory the device (on a shared-memory part, the host) cannot use for anything else. device::release_cached_memory(device) returns it to the driver or the OS: deferred reservations drain, parked buffers retire and empty blocks release, the same three steps the runtime takes before it reports out-of-memory. Memory still in use, or whose last use has not completed on the device, is never touched; a later call can release more once that work retires. Automatic placement calls it on a failed accelerator before the CPU fallback loads.

release_cached.cpp
#include <cstdio>

#include <ClikaRT/clika_rt.h>

using ClikaRT::DataType;
using ClikaRT::Device;
using ClikaRT::Tensor;

static void report(const char* when) {
    const ClikaRT::device::MemoryStats s =
        ClikaRT::device::memory_stats(Device::cpu());
    std::printf("%-14s active %8.1f MiB, cached %8.1f MiB\n", when,
                s.active_bytes / 1048576.0, s.cached_bytes / 1048576.0);
}

int main() {
    {
        // 256 MiB of Float32 work: the pool reserves real memory for it.
        Tensor big = Tensor::zeros({64, 1024, 1024}, DataType::Float32);
        big.synchronize();
        report("in use:");
    }
    // The tensor is gone, but the pool keeps its bytes idle for reuse.
    report("dropped:");

    // Hand the idle reserve back to the system. Memory still in use, or
    // whose last use has not completed, is never touched.
    ClikaRT::device::release_cached_memory(Device::cpu());
    report("released:");
    return 0;
}
in use:        active    256.0 MiB, cached      0.0 MiB
dropped:       active      0.0 MiB, cached    256.0 MiB
released:      active      0.0 MiB, cached      0.0 MiB

The C table carries the same member as release_cached_memory, and the Go, Kotlin and Rust bindings expose it through their generated tiers.

The bundle's compute example covers device movement and this wrap in its 03_data_movement and 06_zero_copy chapters; the custom-operator guide uses from_blob to hand a hand-written kernel's output back to the runtime.