ClikaRT::ops::geglu

function

geglu()

Tensor geglu(Tensor x, GeluMode approximate = GeluMode::None)

GeGLU gated activation over a concatenated gate‖up tensor.

$$\mathrm{geglu}(x)_{\ldots,j} = \mathrm{gelu}(a_{\ldots,j})\cdot b_{\ldots,j},\qquad x = [\,a \,\|\, b\,] \text{ along the last dim}$$

The input's LAST dimension must be even (2d); the output halves it (d). approximate selects the exact (erf) or tanh GELU for the gate.

Parameters

• x: [.., 2d]; gate a first, up b second.
• approximate: GeluMode::None = exact erf form (default); GeluMode::Tanh = the tanh approximation.

Throws

• ClikaRT::Error: when the last dimension's size is odd, or when the input's dtype is not served (code_name() carries the reason).

Returns: [.., d] (the input shape with the last dim halved); dtype of x.

// projection emits [tokens, 2*ffn]; geglu gates down to [tokens, ffn]:
auto h = ClikaRT::ops::geglu(proj);

Declared in ClikaRT/compute/ops.h, line 741