NNlib

celu(x, α=1) =
    (x ≥ 0 ? x : α * (exp(x/α) - 1))

Continuously Differentiable Exponential Linear Units See Continuously Differentiable Exponential Linear Units.

elu(x, α=1) =
  x > 0 ? x : α * (exp(x) - 1)

Exponential Linear Unit activation function. See Fast and Accurate Deep Network Learning by Exponential Linear Units. You can also specify the coefficient explicitly, e.g. elu(x, 1).

gelu(x) = 0.5x * (1 + tanh(√(2/π) * (x + 0.044715x^3)))

Gaussian Error Linear Unit activation function.

hardσ(x, a=0.2) = max(0, min(1.0, a * x + 0.5))

Segment-wise linear approximation of sigmoid. See BinaryConnect: Training Deep Neural Networks withbinary weights during propagations.

hardtanh(x) = max(-1, min(1, x))

Segment-wise linear approximation of tanh. Cheaper and more computational efficient version of tanh. See Large Scale Machine Learning.

leakyrelu(x, a=0.01) = max(a*x, x)

Leaky Rectified Linear Unit activation function. You can also specify the coefficient explicitly, e.g. leakyrelu(x, 0.01).

lisht(x) = x * tanh(x)

Non-Parametric Linearly Scaled Hyperbolic Tangent Activation Function. See LiSHT

logcosh(x)

Return log(cosh(x)) which is computed in a numerically stable way.

logσ(x)

Return log(σ(x)) which is computed in a numerically stable way.

julia> logσ(0)
-0.6931471805599453
julia> logσ.([-100, -10, 100])
3-element Array{Float64,1}:
 -100.0
  -10.000045398899218
   -3.720075976020836e-44

mish(x) = x * tanh(softplus(x))

Self Regularized Non-Monotonic Neural Activation Function. See Mish: A Self Regularized Non-Monotonic Neural Activation Function.

relu(x) = max(0, x)

Rectified Linear Unit activation function.

relu6(x) = min(max(0, x), 6)

Rectified Linear Unit activation function capped at 6. See Convolutional Deep Belief Networks on CIFAR-10

rrelu(x, l=1/8, u=1/3) = max(a*x, x)

a = randomly sampled from uniform distribution U(l, u)

Randomized Leaky Rectified Linear Unit activation function. You can also specify the bound explicitly, e.g. rrelu(x, 0.0, 1.0).

selu(x) = λ * (x ≥ 0 ? x : α * (exp(x) - 1))

λ ≈ 1.0507
α ≈ 1.6733

Scaled exponential linear units. See Self-Normalizing Neural Networks.

σ(x) = 1 / (1 + exp(-x))

Classic sigmoid activation function.

softplus(x) = log(exp(x) + 1)

See Deep Sparse Rectifier Neural Networks.

softshrink(x, λ=0.5) =
    (x ≥ λ ? x - λ : (-λ ≥ x ? x + λ : 0))

See Softshrink Activation Function.

softsign(x) = x / (1 + |x|)

See Quadratic Polynomials Learn Better Image Features.

swish(x) = x * σ(x)

Self-gated activation function. See Swish: a Self-Gated Activation Function.

tanhshrink(x) = x - tanh(x)

See Tanhshrink Activation Function.

trelu(x, theta = 1.0) = x > theta ? x : 0

Threshold Gated Rectified Linear. See ThresholdRelu

softmax(x; dims=1)

Softmax turns input array x into probability distributions that sum to 1 along the dimensions specified by dims. It is semantically equivalent to the following:

softmax(x; dims=1) = exp.(x) ./ sum(exp.(x), dims=dims)

with additional manipulations enhancing numerical stability.

For a matrix input x it will by default (dims=1) treat it as a batch of vectors, with each column independent. Keyword dims=2 will instead treat rows independently, etc...

julia> softmax([1, 2, 3])
3-element Array{Float64,1}:
  0.0900306
  0.244728
  0.665241

See also logsoftmax.

logsoftmax(x; dims=1)

Computes the log of softmax in a more numerically stable way than directly taking log.(softmax(xs)). Commonly used in computing cross entropy loss.

It is semantically equivalent to the following:

logsoftmax(x; dims=1) = x .- log.(sum(exp.(x), dims=dims))

See also softmax.

maxpool(x, k::NTuple; pad=0, stride=k)

Perform max pool operation with window size k on input tensor x.

meanpool(x, k::NTuple; pad=0, stride=k)

Perform mean pool operation with window size k on input tensor x.

conv(x, w; stride=1, pad=0, dilation=1, flipped=false)

Apply convolution filter w to input x. x and w are 3d/4d/5d tensors in 1d/2d/3d convolutions respectively.

depthwiseconv(x, w; stride=1, pad=0, dilation=1, flipped=false)

Depthwise convolution operation with filter w on input x. x and w are 3d/4d/5d tensors in 1d/2d/3d convolutions respectively.

batched_mul(A, B) -> C

Batched matrix multiplication. Result has C[:,:,k] == A[:,:,k] * B[:,:,k] for all k.

batched_mul!(C, A, B) -> C

In-place batched matrix multiplication, equivalent to mul!(C[:,:,k], A[:,:,k], B[:,:,k]) for all k.

batched_transpose(A::AbstractArray{T,3})
batched_adjoint(A)

Equivalent to applying transpose or adjoint to each matrix A[:,:,k].

These exist to control how batched_mul behaves, as it operated on such matrix slices of an array with ndims(A)==3.

BatchedTranspose{T, N, S} <: AbstractBatchedMatrix{T, N}
BatchedAdjoint{T, N, S}

Lazy wrappers analogous to Transpose and Adjoint, returned by batched_transpose.

batched_transpose(A::AbstractArray{T,3})
batched_adjoint(A)

Equivalent to applying transpose or adjoint to each matrix A[:,:,k].

These exist to control how batched_mul behaves, as it operated on such matrix slices of an array with ndims(A)==3.

BatchedTranspose{T, N, S} <: AbstractBatchedMatrix{T, N}
BatchedAdjoint{T, N, S}

Lazy wrappers analogous to Transpose and Adjoint, returned by batched_transpose.