A Gaussian Error Linear Unit (GELU) is an activation function that maps x into x*G(x), where G is the cumulative distribution function of the standard Gaussian (with mean 0 and standard deviation 1).