Gaussian error linear units (gelus)
D Hendrycks, K Gimpel - arXiv preprint arXiv:1606.08415, 2016 - arxiv.org
arXiv preprint arXiv:1606.08415, 2016•arxiv.org
We propose the Gaussian Error Linear Unit (GELU), a high-performing neural network
activation function. The GELU activation function is $ x\Phi (x) $, where $\Phi (x) $ the
standard Gaussian cumulative distribution function. The GELU nonlinearity weights inputs
by their value, rather than gates inputs by their sign as in ReLUs ($ x\mathbf {1} _ {x> 0} $).
We perform an empirical evaluation of the GELU nonlinearity against the ReLU and ELU
activations and find performance improvements across all considered computer vision …
activation function. The GELU activation function is $ x\Phi (x) $, where $\Phi (x) $ the
standard Gaussian cumulative distribution function. The GELU nonlinearity weights inputs
by their value, rather than gates inputs by their sign as in ReLUs ($ x\mathbf {1} _ {x> 0} $).
We perform an empirical evaluation of the GELU nonlinearity against the ReLU and ELU
activations and find performance improvements across all considered computer vision …
We propose the Gaussian Error Linear Unit (GELU), a high-performing neural network activation function. The GELU activation function is , where the standard Gaussian cumulative distribution function. The GELU nonlinearity weights inputs by their value, rather than gates inputs by their sign as in ReLUs (). We perform an empirical evaluation of the GELU nonlinearity against the ReLU and ELU activations and find performance improvements across all considered computer vision, natural language processing, and speech tasks.
arxiv.org
