Skip to main content

SwiGLU

SwiGLU(alpha=1.0, beta=0.0, limit=float("inf"))

SwiGLU over a concatenated [*, 2d] gate-then-up input, emitting [*, d]:
(clamp(up, -limit, limit) + beta) * G * sigmoid(alpha * G) with G =
min(gate, limit); the defaults reduce to silu(gate) * up.

The module form of ``clika_runtime.nn.functional.swiglu``: construct it
once with the settings, then call it like a function.

Args:
    alpha: the sigmoid sharpness on the gate half.
    beta: added to the clamped up half.
    limit: the clamp bound on both halves; infinity disables the clamp.

__init__​

__init__(self, alpha: 'float' = 1.0, beta: 'float' = 0.0, limit: 'float' = inf) -> 'None'

Initialize self. See help(type(self)) for accurate signature.

extra_repr​

extra_repr(self) -> 'str'

extra_repr() -> str

One line of per-class detail for :meth:__repr__; a layer prints its geometry here (in_features=64, out_features=256).

forward​

forward(self, input: 'Tensor') -> 'Tensor'

forward(input) -> Tensor

Runs clika_runtime.nn.functional.swiglu on input with the stored settings; returns a new tensor.