Cyclic schedule API reference

CosAnneal(l0, l1, period, restart = true)
CosAnneal(; l0, l1, period, restart = true)

A cosine annealing schedule (see "SGDR: Stochastic Gradient Descent with Warm Restarts") The output conforms to

t̂ = restart ? mod(t - 1, period) : (t - 1)
abs(l0 - l1) * (1 + cos(π * t̂ / period)) / 2 + min(l0, l1)

This schedule is also referred to as "cosine annealing (with warm restarts)" in machine learning literature.

Arguments

• range == abs(l0 - l1): the dynamic range (given by the endpoints)
• offset == min(l0, l1): the offset / minimum value
• period::Integer: the period
• restart::Bool: use warm-restarts

Sin(l0, l1, period)
Sin(; l0, l1, period)

A sine wave schedule with period. The output conforms to

abs(l0 - l1) * abs(sin(π * (t - 1) / period)) + min(l0, l1)

Arguments

• range == abs(l0 - l1): the dynamic range (given by the endpoints)
• offset == min(l0, l1): the offset / minimum value
• period::Integer: the period

Triangle{T, S<:Integer}(l0, l1, period)
Triangle(; l0, l1, period)

A triangle wave schedule with period. The output conforms to

abs(l0 - l1) * (2 / π) * abs(asin(sin(π * (t - 1) / period))) + min(l0, l1)

Arguments

• range == abs(l0 - l1): the dynamic range (given by the endpoints)
• offset == min(l0, l1): the offset / minimum value
• period::Integer: the period

SinDecay2(l0, l1, period)
SinDecay2(; l0, l1, period)

A sine wave schedule with period and half the amplitude each cycle. The output conforms to

abs(l0 - l1) * Sin(t) / (2^floor((t - 1) / period)) + min(l0, l1)

where Sin(t) is abs(sin(π * (t - 1) / period)) (see Sin).

Arguments

• range == abs(l0 - l1): the dynamic range (given by the endpoints)
• offset == min(l0, l1): the offset / minimum value
• period::Integer: the period

SinExp(l0, l1, period, decay)
SinExp(; l0, l1, period, decay)

A sine wave schedule with period and an exponentially decaying amplitude. The output conforms to

abs(l0 - l1) * Sin(t) * γ^(t - 1) + min(l0, l1)

where Sin(t) is abs(sin(π * (t - 1) / period)) (see Sin).

Arguments

• range == abs(l0 - l1): the dynamic range (given by the endpoints)
• offset == min(l0, l1): the offset / minimum value
• period::Integer: the period
• decay: the decay rate

TriangleDecay2{T, S<:Integer}(l0, l1, period)
TriangleDecay2(; l0, l1, period)

A triangle wave schedule with period and half the amplitude each cycle. The output conforms to

abs(l0 - l1) * Triangle(t) / (2^floor((t - 1) / period)) + min(l0, l1)

where Triangle(t) is (2 / π) * abs(asin(sin(π * (t - 1) / schedule.period))) (see Triangle).

Arguments

• range == abs(l0 - l1): the dynamic range (given by the endpoints)
• offset == min(l0, l1): the offset / minimum value
• period::Integer: the period

TriangleExp{T, S<:Integer}(l0, l1, period, decay)
TriangleExp(; l0, l1, period, decay)

A triangle wave schedule with period and an exponentially decaying amplitude. The output conforms to

abs(l0 - l1) * Triangle(t) * decay^(t - 1) + min(l0, l1)

where Triangle(t) is (2 / π) * abs(asin(sin(π * (t - 1) / schedule.period))) (see Triangle).

Arguments

• range == abs(l0 - l1): the dynamic range (given by the endpoints)
• offset == min(l0, l1): the offset / minimum value
• period::Integer: the period
• decay: the decay rate