ParameterSchedulers.jl provides common machine learning (ML) schedulers for hyper-parameters. Though this package is framework agnostic, a convenient interface for pairing schedules with Flux.jl optimizers is available. Using this package with Flux is as simple as:
using Flux, ParameterSchedulers
using ParameterSchedulers: Scheduler
opt = Scheduler(Momentum, Exp(start = 1e-2, decay = 0.8))
This is a table of the common schedules implemented, but ParameterSchedulers provides utilities for creating more exotic schedules as well. The higher order schedules should make it so that you will rarely need to write a schedule from scratch.
You can read this paper for more information on the schedules below.
| Schedule |
Description |
Type |
Example |
|
Step(; start, decay, step_sizes)
|
Exponential decay by decay every step in step_sizes
|
Decay |
+---------------+
|⠉⠉⢆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀|
|⠀⠀⠀⠉⠉⠉⠉⢢⠀⠀⠀⠀⠀⠀⠀|
|⠀⠀⠀⠀⠀⠀⠀⠀⠉⠉⠑⢄⣀⣀⣀|
+---------------+
|
|
Exp(start, decay)
|
Exponential decay by decay every iteration
|
Decay |
+---------------+
|⢣⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀|
|⠀⠣⢄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀|
|⠀⠀⠀⠓⠢⠤⢄⣀⣀⣀⣀⣀⣀⣀⣀|
+---------------+
|
|
CosAnneal(;l0, l1, period)
|
Cosine annealing
|
Cyclic |
+---------------+
|⠉⠒⡄⠀⠀⠀⡎⠑⠢⡀⠀⠀⢠⠋⠒|
|⠀⠀⠈⢆⠀⢰⠁⠀⠀⠑⡄⠀⡜⠀⠀|
|⠀⠀⠀⠀⠣⡇⠀⠀⠀⠀⠈⢶⠁⠀⠀|
+---------------+
|
|
OneCycle(nsteps, maxval)
|
One cycle cosine
|
Complex |
+---------------+
|⠀⠀⠀⡤⠊⠉⠉⠒⠤⡀⠀⠀⠀⠀⠀|
|⠀⢀⠜⠀⠀⠀⠀⠀⠀⠈⠑⢄⠀⠀⠀|
|⡠⠊⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠢⢄|
+---------------+
|
|
Triangle(l0, l1, period)
|
Triangle wave function
|
Cyclic |
+---------------+
|⠀⡼⡀⠀⡸⡇⠀⢠⢣⠀⠀⡾⡀⠀⡸|
|⢰⠁⢇⢀⠇⢸⠀⡜⠈⡆⢸⠀⢣⢀⠇|
|⡎⠀⠘⡜⠀⠀⢷⠁⠀⢱⡇⠀⠘⡜⠀|
+---------------+
|
|
TriangleDecay2(l0, l1, period)
|
Triangle wave function with half the amplitude every period
|
Cyclic |
+---------------+
|⠀⡼⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀|
|⢰⠁⢇⠀⢠⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀|
|⡎⠀⠘⣤⠃⠘⣄⡠⠒⢄⣀⡠⢄⣀⣀|
+---------------+
|
|
TriangleExp(l0, l1, period, decay)
|
Triangle wave function with exponential amplitude decay at rate decay
|
Cyclic |
+---------------+
|⠀⡼⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀|
|⢰⠁⢇⠀⡜⢇⠀⠀⡀⠀⠀⠀⠀⠀⠀|
|⡎⠀⠘⡼⠀⠈⢆⠎⠈⢆⣀⠜⠢⣀⠤|
+---------------+
|
|
Poly(start, degree, max_iter)
|
Polynomial decay at degree degree.
|
Decay |
+---------------+
|⠉⠢⢄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀|
|⠀⠀⠀⠉⠒⠤⣀⠀⠀⠀⠀⠀⠀⠀⠀|
|⠀⠀⠀⠀⠀⠀⠀⠉⠓⠒⠤⠤⣀⣀⣀|
+---------------+
|
|
Inv(start, decay, degree)
|
Inverse decay at rate (1 + t * decay)^degree
|
Decay |
+---------------+
|⢇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀|
|⠈⡆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀|
|⠀⠈⠑⠤⠤⢄⣀⣀⣀⣀⣀⣀⣀⣀⣀|
+---------------+
|
|
Sin(;l0, l1, period)
|
Sine function
|
Cyclic |
+---------------+
|⠀⡼⡀⠀⡸⡇⠀⢠⢣⠀⠀⡾⡀⠀⡸|
|⢰⠁⢇⢀⠇⢸⠀⡜⠈⡆⢸⠀⢣⢀⠇|
|⡎⠀⠘⡜⠀⠀⢷⠁⠀⢱⡇⠀⠘⡜⠀|
+---------------+
|
|
SinDecay2(l0, l1, period)
|
Sine function with half the amplitude every period
|
Cyclic |
+---------------+
|⠀⡼⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀|
|⢰⠁⢇⠀⡠⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀|
|⡎⠀⠘⡴⠁⠘⣄⡠⠒⢄⣀⡠⢄⣀⣀|
+---------------+
|
|
SinExp(l0, l1, period)
|
Sine function with exponential amplitude decay at rate decay
|
Cyclic |
+---------------+
|⠀⡼⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀|
|⢰⠁⢇⠀⡜⢇⠀⠀⡀⠀⠀⠀⠀⠀⠀|
|⡎⠀⠘⡼⠀⠈⢆⠎⠈⢆⣀⠜⠢⣀⠤|
+---------------+
|