Skip to main content

ClikaRT::nn::Module

class

Header: ClikaRT/nn/module.h
Inherits: std::enable_shared_from_this<Module>
Inherited by: ClikaRT::nn::Conv, ClikaRT::nn::ConvTranspose, ClikaRT::nn::Embedding, ClikaRT::nn::LayerNorm, ClikaRT::nn::Linear, ClikaRT::nn::MoE, ClikaRT::nn::QConv, ClikaRT::nn::QConvWoQ, ClikaRT::nn::QMoEWoQ, ClikaRT::nn::RMSNorm

Base class of every ClikaRT module: an ownership tree of parameters, buffers, and child modules.

A module registers its tensors (register_parameter / register_buffer) and children (register_module); load_state_dict binds a checkpoint onto the registered names; the first forward (or initialize()) packs the weights into their backend form; after the pack exactly ONE resident copy of each weight exists. to(device/dtype) moves or casts the tree; cpu() is the host shorthand. Held by std::shared_ptr (module trees share children).

Member functions​

~Module()​

virtual ~Module()

Virtual base destructor; derived modules release their packed weights.

Declared in ClikaRT/nn/module.h, line 137

Module(Module)​

Module(const Module&) =delete

Declared in ClikaRT/nn/module.h, line 139

operator=(Module)​

Module& operator=(const Module&) =delete

Declared in ClikaRT/nn/module.h, line 140

Module(Module)​

Module(Module&&) =delete

Declared in ClikaRT/nn/module.h, line 141

operator=(Module)​

Module& operator=(Module&&) =delete

Declared in ClikaRT/nn/module.h, line 142

register_parameter(string_view, Tensor)​

Tensor register_parameter(std::string_view name, Tensor tensor)

Register tensor as a learnable parameter under name; returns the registered handle (shares tensor's storage).

Declared in ClikaRT/nn/module.h, line 151

register_buffer(string_view, Tensor)​

Tensor register_buffer(std::string_view name, Tensor tensor)

Register tensor as a non-learnable buffer under name; returns the registered handle (shares tensor's storage).

Declared in ClikaRT/nn/module.h, line 154

register_parameter(string_view, FakeTensor)​

Tensor register_parameter(std::string_view name, const FakeTensor& spec)

Declare a storage-free SLOT from a metadata spec (shape / dtype / placement): the slot enumerates under its dotted name, so load_state_dict matches it and binds the checkpoint tensor onto the spec's placement + dtype, and reads as Tensor::is_fake() until bound. A module built this way holds no weight memory until loading. The returned handle is the registered slot; re-registering the same name (e.g. with a real tensor) replaces it, keeping its position.

Declared in ClikaRT/nn/module.h, line 162

register_buffer(string_view, FakeTensor)​

Tensor register_buffer(std::string_view name, const FakeTensor& spec)

Declare a storage-free buffer slot under name from the metadata spec; a later bind fills it (the buffer twin of the parameter overload above).

Declared in ClikaRT/nn/module.h, line 166

register_module()​

void register_module(std::string_view name, std::shared_ptr<Module> module)

Attach a child module under name (the tree edge name.child_key names its tensors).

Declared in ClikaRT/nn/module.h, line 169

named_parameters()​

std::vector<std::pair<std::string, Tensor>> named_parameters(bool recurse = true) const

All parameters as (dotted-name, tensor) pairs; recurse walks children. A tensor shared under several names appears once, under the first. Parameters a layer holds in packed runtime form (a Linear weight after initialize() or the first forward) are included as on-demand reconstructions; see packed_parameters().

Declared in ClikaRT/nn/module.h, line 180

packed_parameters()​

virtual std::vector<std::pair<std::string, Tensor>> packed_parameters() const

The parameters THIS module currently holds in packed runtime form, reconstructed on demand as (name, tensor) pairs (names relative to this module, no child recursion). Each returned tensor is a fresh owning copy read back from the packed representation, value-exact, and the module keeps exactly one resident copy (the packed form) throughout. named_parameters() / state_dict() merge these under their dotted names, so a checkpoint taken after packing is complete. The base returns empty; packing layers override. A layer posture whose packed form has no reconstruction yet (a quantized pack, a fused projection group) contributes nothing here; its checkpoint is taken before the first forward.

Declared in ClikaRT/nn/module.h, line 193

named_buffers()​

std::vector<std::pair<std::string, Tensor>> named_buffers(bool recurse = true) const

All buffers as (dotted-name, tensor) pairs; recurse walks children.

Declared in ClikaRT/nn/module.h, line 196

named_children()​

std::vector<std::pair<std::string, std::shared_ptr<Module>>> named_children() const

Direct children as (name, module) pairs.

Declared in ClikaRT/nn/module.h, line 198

parameters()​

std::vector<Tensor> parameters(bool recurse = true) const

All parameter tensors (names dropped); recurse walks children.

Declared in ClikaRT/nn/module.h, line 201

buffers()​

std::vector<Tensor> buffers(bool recurse = true) const

All buffer tensors (names dropped); recurse walks children.

Declared in ClikaRT/nn/module.h, line 204

state_dict()​

NamedTensors state_dict(bool recurse = true) const

The module's checkpoint view: every named parameter and named buffer under its dotted name, the load_state_dict dual: a dict taken here loads back exactly (the asserted round-trip contract). A weight a layer has packed for serving is reconstructed on demand (see packed_parameters()), so the dict stays complete after initialize() or a forward; the reconstructed entries are fresh owning copies, while still-bound entries share storage with the module. A layer posture with no reconstruction yet (a quantized pack, a fused projection group) keeps its packed entries out of a post-forward dict; take the checkpoint before the first forward there.

Declared in ClikaRT/nn/module.h, line 216

to(StreamOrDevice)​

void to(StreamOrDevice where)

Move the tree to a placement: a Device, or a Stream so the tree computes on that stream and lands its outputs there (the lane-placement call: build/load once, to(lane) once, forward every step). Children move first; a packed leaf (Linear / QLinearWoQ / MoE / QMoEWoQ) rebuilds its pack on the target, and a still-fake slot re-declares its metadata there.

Throws

  • ClikaRT::Error: when the tree cannot move as asked.

Declared in ClikaRT/nn/module.h, line 244

to(DataType)​

void to(DataType dtype)

Cast the tree's parameters/buffers to dtype (children first). A dense packed leaf restores its raw weights, casts, and re-packs on the next forward; quantized leaves refuse dtype moves.

Throws

  • ClikaRT::Error: when the cast cannot be served (a quantized leaf, e.g.).

Declared in ClikaRT/nn/module.h, line 250

load_state_dict(NamedTensors, LoadOptions)​

void load_state_dict(const NamedTensors& checkpoint, LoadOptions options = {})

Bind every parameter/buffer whose dotted name is present in checkpoint (placed on the slot's declared device + dtype). With options.strict, raises ClikaRT::Error if any declared name is missing from the checkpoint or any checkpoint key is unexpected, minus the per-name allowances in options, the same completeness verdict the container overload below applies. Note: a bare braced second argument (load_state_dict(ckpt, {…})) is ambiguous against the strictness-only overload below; spell LoadOptions{…} (or a named local) explicitly.

Declared in ClikaRT/nn/module.h, line 263

load_state_dict(NamedTensors, bool)​

void load_state_dict(const NamedTensors& checkpoint, bool strict)

Bind every parameter/buffer whose dotted name is present in checkpoint (placed on the slot's declared device + dtype). With options.strict, raises ClikaRT::Error if any declared name is missing from the checkpoint or any checkpoint key is unexpected, minus the per-name allowances in options, the same completeness verdict the container overload below applies. Note: a bare braced second argument (load_state_dict(ckpt, {…})) is ambiguous against the strictness-only overload below; spell LoadOptions{…} (or a named local) explicitly.

Declared in ClikaRT/nn/module.h, line 269

load_state_dict(io::TensorsContainer, LoadOptions)​

void load_state_dict(io::TensorsContainer& checkpoint, LoadOptions options = {})

Bind every parameter/buffer whose dotted name is present in checkpoint (placed on the slot's declared device + dtype). With options.strict, raises ClikaRT::Error if any declared name is missing from the checkpoint or any checkpoint key is unexpected, minus the per-name allowances in options, the same completeness verdict the container overload below applies. Note: a bare braced second argument (load_state_dict(ckpt, {…})) is ambiguous against the strictness-only overload below; spell LoadOptions{…} (or a named local) explicitly.

Declared in ClikaRT/nn/module.h, line 285

initialize()​

void initialize()

Run the one-time weight pack NOW instead of on the first forward (every packing leaf packs lazily on first use, once, thread-safe; this warm-up lets a serving process pay every pack at load time rather than on the first token). Idempotent. The base recurses into children (registration order), so ONE call on the root warms the whole tree; a leaf with a packable form overrides initialize_impl() with its own pack; a fusing parent overrides it and deliberately does not recurse into the children it consumed.

Throws

  • ClikaRT::Error: when a pack fails (e.g. a weight slot that was never loaded).

Declared in ClikaRT/nn/module.h, line 299

evict_weights()​

void evict_weights()

Release every disk-backed weight's memory to its file backing (registered parameters/buffers and the packed weights the one-time pack produced), recursively over the whole tree. A weight that is not disk-backed is untouched; the next use restores an evicted weight transparently. Meaningful for models loaded with a disk-backed policy (a sticky lazy checkpoint); requires no forward to be in flight.

Declared in ClikaRT/nn/module.h, line 309

prefetch_weights()​

void prefetch_weights()

Warm every disk-backed weight ahead of use (read-ahead on resident backings; an evicted weight restores now), the inverse of evict_weights, recursively over the whole tree.

Declared in ClikaRT/nn/module.h, line 315

output_shapes()​

virtual std::vector<FakeTensor> output_shapes(ClikaRT::Span<const FakeTensor>) const

Shape rule: read input metadata (each input a FakeTensor: SymInt dims, dtype, placement, quantized flag) → output FakeTensors. Compose dimensions with SymInt / SymFloat; declare a data-dependent extent with SymInt::unknown(...). Outputs are placed on the op's stream (derived from the inputs), so the output FakeTensor's placement is informational. An exception thrown here surfaces as the dispatch call's error (a ClikaRT::Error keeps its Status; any other exception reports Status::Internal). Default: an error (a composite module never calls dispatch, so it need not override this).

Declared in ClikaRT/nn/module.h, line 327

compute()​

virtual void compute(ClikaRT::Span<const Tensor>, ClikaRT::Span<Tensor>) const

Kernel: outputs arrive pre-allocated on the op's device (sharing storage). Read inputs and write outputs through the pointers that match where the kernel runs. device_const_data_ptr() / device_mutable_data_ptr() are for a kernel on the op's device: no host wait, and the pointer is ordered on the op's stream. const_data_ptr() / mutable_data_ptr() are for a plain host loop over CPU-resident tensors: they wait for the data and refuse a device tensor. The runtime binds the op's stream (read it with this->stream()) and makes its device current before calling this, so a GPU kernel launches on the right device.

This is the hand-written-kernel hook. compute() runs on whatever device its tensors live on; branch on this->stream().device() and launch directly:

Stream s = this->stream();                                     // the op's stream
auto cu = static_cast<cudaStream_t>(s.native_handle());        // the queue
const float* q = static_cast<const float*>(inputs[0].device_const_data_ptr());
float*       o = static_cast<float*>(outputs[0].device_mutable_data_ptr());
my_attention_kernel<<<grid, block, smem, cu>>>(q, ..., o);     // your kernel

Size the launch from get_device_properties(this->stream().device()) (SM count, shared-mem budget); take stream-ordered scratch from this->stream().allocate(shape, dtype). An exception thrown here does not escape: it surfaces as the operation's error at the caller's synchronization point (a ClikaRT::Error keeps its Status; any other exception reports Status::Internal), and the process keeps running. Default: an error.

Declared in ClikaRT/nn/module.h, line 362

stream()​

Stream stream() const

The stream the runtime is currently dispatching this module on; valid ONLY inside compute() (and anything it calls). It is the queue a hand-written kernel launches on (stream().native_handle()) and the device the op runs on (stream().device()). Calling it outside a compute() invocation raises ClikaRT::Error.

Declared in ClikaRT/nn/module.h, line 375

dispatch()​

std::vector<Tensor> dispatch(
    ClikaRT::Span<const Tensor> inputs,
    Device device = Device::cpu()
) const

Dispatch THIS module's (output_shapes, compute) through the runtime: eager runs compute; tracing records the op from output_shapes alone. device (default CPU) picks where a leaf with no tensor inputs runs.

Throws

  • ClikaRT::Error: when the dispatch is rejected (e.g. output_shapes reports an error).

Declared in ClikaRT/nn/module.h, line 383

Protected member functions​

Module()​

Module()

Constructs the module and captures the hook entry points compiled into the translation unit this constructor runs in (the unit that defines the most derived constructor). The runtime reaches every overridable hook (output_shapes, compute, to_impl, initialize_impl, packed_parameters, apply_weight_residency) through those entry points, so an exception an override raises is caught by the same C++ runtime that raised it and arrives in the library as a status; no exception unwinds across the library boundary.

Declared in ClikaRT/nn/module.h, line 394

Module(ModuleHooks)​

explicit Module(const ModuleHooks& hooks)

The constructor Module() delegates to, with the hook entry points.

Declared in ClikaRT/nn/module.h, line 396

apply_weight_residency()​

virtual void apply_weight_residency(
    WeightResidency residency,
    std::uint64_t& stamped,
    std::uint64_t& class_pinned
)

The load-scope residency stamp (LoadOptions::weight_residency routes here before the checkpoint binds). The base recurses into children in registration order; a module that admits a residency choice overrides this to adopt residency (counting itself in stamped) or to keep a class-pinned posture (counting itself in class_pinned), and then calls the base so its own children are still walked. A subclass with no residency of its own needs no override.

Declared in ClikaRT/nn/module.h, line 405