ClikaRT::nn::LayerNorm

class

Header: ClikaRT/nn/layer_norm.h
Inherits: ClikaRT::nn::Module

A bound LayerNorm, the public face of the runtime's ops::layer_norm, exposed as an nn::Module leaf. Unlike the stateless ops::layer_norm free function, a LayerNorm binds its weight (scale) / bias ONCE and materializes them contiguous at initialize(), so every forward reuses the bound params; build one per norm site at load, reuse it every step.

The scale / bias stay in their OWN dtype (a fp32 scale against a bf16 activation is read at full precision, never cast at rest); initialize() does layout work only. The output dtype follows the input x.

The lifecycle (uniform across every weight-bearing nn module): make(normalized_size, ...) registers storage-free weight / bias slots under their canonical names; set_weights(...) (positional) or load_state_dict(...) (a checkpoint, by dotted name) binds them; the FIRST forward materializes contiguous (once, thread-safe). initialize() remains an optional warm-up.

Held via std::shared_ptr (an nn::Module leaf); copy/move are pinned by the base.

Layer normalization module.

$$y = \frac{x - \mathrm{E}[x]}{\sqrt{\mathrm{Var}[x] + \varepsilon}} \cdot \gamma + \beta$$

Statistics are computed over the trailing normalized_shape dims, the module form of ops::layer_norm, holding gamma/beta as its weights. The module owns its weights: construct with make(...), or declare shapes and bind a checkpoint via load_state_dict. After the first forward (or initialize()) the weight lives ONLY in the backend's packed form (one resident copy); to(dtype) restores, casts, and repacks on the next forward.

auto ln = ClikaRT::nn::LayerNorm::make(gamma, beta);
auto y  = ln->forward(x);

Static member functions

make()

static std::shared_ptr<LayerNorm> make(
    std::int64_t normalized_size,
    bool bias = true,
    double eps = 1e-5,
    std::optional<ops::Activation> activation = std::nullopt,
    DataType dtype = DataType::Float32,
    Device device = Device::cpu()
)

Construct from config with storage-free slots: registers a shape-only weight [normalized_size] (and, with bias, a bias [normalized_size]) parameter on dtype/device; no storage is allocated. named_parameters() reports the slots, so a strict load_state_dict both expects and satisfies them; the first forward builds. Raises ClikaRT::Error on a bad normalized size.

Declared in ClikaRT/nn/layer_norm.h, line 64

Member functions

set_weights()

void set_weights(Tensor weight, OptionalTensor bias = {})

Bind the declared slots positionally: weight (scale, [normalized_size]) and, when the module was made with one, bias. The declared placement wins; shapes must match the declaration. Re-binding drops the built primitive; the next forward rebuilds. Raises ClikaRT::Error on a geometry mismatch or a bias without a declared bias slot.

Declared in ClikaRT/nn/layer_norm.h, line 76

~LayerNorm()

~LayerNorm() override

Declared in ClikaRT/nn/layer_norm.h, line 80

initialize_impl()

virtual Result<void> initialize_impl() override

Optional warm-up: materialize the bound weight / bias slots contiguous NOW (layout-only, dtype-preserving; idempotent, thread-safe). Every declared slot must hold a real (loaded) tensor; a still-fake slot is a clean error.

Declared in ClikaRT/nn/layer_norm.h, line 86

to_impl(StreamOrDevice)

virtual Result<void> to_impl(StreamOrDevice where) override

Move to a placement (Device / Stream) or cast to a dtype. Rebuilds the bound primitive on the target after moving the registry slots.

Declared in ClikaRT/nn/layer_norm.h, line 90

to_impl(DataType)

virtual Result<void> to_impl(DataType dtype) override

Declared in ClikaRT/nn/layer_norm.h, line 91

forward()

Tensor forward(Tensor x) const

out = ACT( (x − mean) · rsqrt(var + eps) · weight + bias ); x [*, N] -> [*, N], output dtype = x's. The first call builds (once, thread-safe); later calls reuse the bound primitive. Raises ClikaRT::Error on a shape/dtype mismatch or a still-fake slot.

Declared in ClikaRT/nn/layer_norm.h, line 98