Skip to main content

ClikaRT::ops::reglu

function

reglu()​

Tensor reglu(Tensor x)

ReGLU gated activation over a concatenated gate‖up input.

reglu(x)…,j=relu(a…,j)⋅b…,j,x=[ a ∥ b ] along the last dim\mathrm{reglu}(x)_{\ldots,j} = \mathrm{relu}(a_{\ldots,j})\cdot b_{\ldots,j},\qquad x = [\,a \,\|\, b\,] \text{ along the last dim}

The relu-gated member of the GLU family, beside geglu / swiglu: [.., 2d] in, [.., d] out; the last dim must be even.

Throws

  • ClikaRT::Error: (INVALID_ARGUMENT) when the last dim is odd (code_name() carries the reason).

Returns: the gated tensor [.., d].

Declared in ClikaRT/compute/ops.h, line 752