NNlib

elu(x, α=1) =
  x > 0 ? x : α * (exp(x) - 1)

Exponential Linear Unit activation function. See Fast and Accurate Deep Network Learning by Exponential Linear Units. You can also specify the coefficient explicitly, e.g. elu(x, 1).

gelu(x) = 0.5x * (1 + tanh(√(2/π) * (x + 0.044715x^3)))

Gaussian Error Linear Unit activation function.

leakyrelu(x, a=0.01) = max(a*x, x)

Leaky Rectified Linear Unit activation function. You can also specify the coefficient explicitly, e.g. leakyrelu(x, 0.01).

logcosh(x)

Return log(cosh(x)) which is computed in a numerically stable way.

logσ(x)

Return log(σ(x)) which is computed in a numerically stable way.

julia> logσ(0)
-0.6931471805599453
julia> logσ.([-100, -10, 100])
3-element Array{Float64,1}:
 -100.0
  -10.000045398899218
   -3.720075976020836e-44

relu(x) = max(0, x)

Rectified Linear Unit activation function.

selu(x) = λ * (x ≥ 0 ? x : α * (exp(x) - 1))

λ ≈ 1.0507
α ≈ 1.6733

Scaled exponential linear units. See Self-Normalizing Neural Networks.

σ(x) = 1 / (1 + exp(-x))

Classic sigmoid activation function.

softplus(x) = log(exp(x) + 1)

See Deep Sparse Rectifier Neural Networks.

softsign(x) = x / (1 + |x|)

See Quadratic Polynomials Learn Better Image Features.

swish(x) = x * σ(x)

Self-gated activation function. See Swish: a Self-Gated Activation Function.

softmax(x; dims=1)

Softmax turns input array x into probability distributions that sum to 1 along the dimensions specified by dims. It is semantically equivalent to the following:

softmax(x; dims=1) = exp.(x) ./ sum(exp.(x), dims=dims)

with additional manipulations enhancing numerical stability.

For a matrix input x it will by default (dims=1) treat it as a batch of vectors, with each column independent. Keyword dims=2 will instead treat rows independently, etc...

julia> softmax([1, 2, 3])
3-element Array{Float64,1}:
  0.0900306
  0.244728
  0.665241

See also logsoftmax.

logsoftmax(x; dims=1)

Computes the log of softmax in a more numerically stable way than directly taking log.(softmax(xs)). Commonly used in computing cross entropy loss.

It is semantically equivalent to the following:

logsoftmax(x; dims=1) = x .- log.(sum(exp.(x), dims=dims))

See also softmax.

maxpool(x, k::NTuple; pad=0, stride=k)

Perform max pool operation with window size k on input tensor x.

meanpool(x, k::NTuple; pad=0, stride=k)

Perform mean pool operation with window size k on input tensor x.

conv(x, w; stride=1, pad=0, dilation=1, flipped=false)

Apply convolution filter w to input x. x and w are 3d/4d/5d tensors in 1d/2d/3d convolutions respectively.

depthwiseconv(x, w; stride=1, pad=0, dilation=1, flipped=false)

Depthwise convolution operation with filter w on input x. x and w are 3d/4d/5d tensors in 1d/2d/3d convolutions respectively.

batched_mul(A, B) -> C

Batched matrix multiplication. Result has C[:,:,k] == A[:,:,k] * B[:,:,k] for all k.

batched_mul!(C, A, B) -> C

In-place batched matrix multiplication, equivalent to mul!(C[:,:,k], A[:,:,k], B[:,:,k]) for all k.

batched_transpose(A::AbstractArray{T,3})
batched_adjoint(A)

Equivalent to applying transpose or adjoint to each matrix A[:,:,k].

These exist to control how batched_mul behaves, as it operated on such matrix slices of an array with ndims(A)==3.

BatchedTranspose{T, N, S} <: AbstractBatchedMatrix{T, N}
BatchedAdjoint{T, N, S}

Lazy wrappers analogous to Transpose and Adjoint, returned by batched_transpose

batched_transpose(A::AbstractArray{T,3})
batched_adjoint(A)

Equivalent to applying transpose or adjoint to each matrix A[:,:,k].

These exist to control how batched_mul behaves, as it operated on such matrix slices of an array with ndims(A)==3.

BatchedTranspose{T, N, S} <: AbstractBatchedMatrix{T, N}
BatchedAdjoint{T, N, S}

Lazy wrappers analogous to Transpose and Adjoint, returned by batched_transpose