Skip to main content

ClikaRT::ops::geglu

function

geglu()​

Tensor geglu(Tensor input, GeluMode approximate = GeluMode::None)

GeGLU gated activation over a concatenated gate‖up tensor.

geglu(x)…,j=gelu(a…,j)⋅b…,j,x=[ a ∥ b ] along the last dim\mathrm{geglu}(x)_{\ldots,j} = \mathrm{gelu}(a_{\ldots,j})\cdot b_{\ldots,j},\qquad x = [\,a \,\|\, b\,] \text{ along the last dim}

The input's LAST dimension must be even (2d); the output halves it (d). approximate selects the exact (erf) or tanh GELU for the gate.

Parameters

  • input: [.., 2d]; gate a first, up b second.
  • approximate: GeluMode::None = exact erf form (default); GeluMode::Tanh = the tanh approximation.

Throws

  • ClikaRT::Error: when the last dimension's size is odd, or when the input's dtype is not served (code_name() carries the reason).

Returns: [.., d] (the input shape with the last dim halved); dtype of input.

// projection emits [tokens, 2*ffn]; geglu gates down to [tokens, ffn]:
auto h = ClikaRT::ops::geglu(proj);

Declared in ClikaRT/compute/ops.h, line 742