ParameterSchedulers.jl

Introduction

Tutorials

Interface

API Reference

ParameterSchedulers

Dev Build Status

ParameterSchedulers.jl provides common machine learning (ML) schedulers for hyper-parameters. Though this package is framework agnostic, a convenient interface for pairing schedules with Flux.jl optimizers is available. Using this package with Flux is as simple as:

using Flux, ParameterSchedulers
using ParameterSchedulers: Scheduler

opt = Scheduler(Exp(λ = 1e-2, γ = 0.8), Momentum())

Available Schedules

This is a table of the common schedules implemented, but ParameterSchedulers provides utilities for creating more exotic schedules as well. You can read this paper for more information on the schedules below.

Schedule Description Type Example

Step(;λ, γ, step_sizes)

Exponential decay by γ every step in step_sizes

 Decay 
    +---------------+ 
    |⠉⠙⢄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀|
    |⠀⠀⠈⠒⠒⠒⠒⡄⠀⠀⠀⠀⠀⠀⠀|
    |⠀⠀⠀⠀⠀⠀⠀⠈⠦⠤⠤⠤⠤⠤⠤|
    +---------------+

Exp(;λ, γ)

Exponential decay by γ every iteration

 Decay 
    +---------------+ 
    |⢣⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀|
    |⠀⠣⢄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀|
    |⠀⠀⠀⠓⠢⠤⢄⣀⣀⣀⣀⣀⣀⣀⣀|
    +---------------+

Poly(;λ, p, max_iter)

Polynomial decay at degree p

 Decay 
    +---------------+ 
    |⠉⠢⢄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀|
    |⠀⠀⠀⠉⠒⠤⣀⠀⠀⠀⠀⠀⠀⠀⠀|
    |⠀⠀⠀⠀⠀⠀⠀⠉⠓⠒⠤⠤⣀⣀⣀|
    +---------------+

Inv(;λ, γ, p)

Inverse decay at rate (1 + tγ)^p

 Decay 
    +---------------+ 
    |⢇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀|
    |⠈⡆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀|
    |⠀⠈⠑⠤⠤⢄⣀⣀⣀⣀⣀⣀⣀⣀⣀|
    +---------------+

Triangle(;λ0, λ1, period)

Triangle wave function

 Cyclic 
    +---------------+ 
    |⠀⡼⡀⠀⡸⡇⠀⢠⢣⠀⠀⡾⡀⠀⡸|
    |⢰⠁⢇⢀⠇⢸⠀⡜⠈⡆⢸⠀⢣⢀⠇|
    |⡎⠀⠘⡜⠀⠀⢷⠁⠀⢱⡇⠀⠘⡜⠀|
    +---------------+

TriangleDecay2(;λ0, λ1, period)

Triangle wave function with half the amplitude every period

 Cyclic 
    +---------------+ 
    |⠀⡼⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀|
    |⢰⠁⢇⠀⢠⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀|
    |⡎⠀⠘⣤⠃⠘⣄⡠⠒⢄⣀⡠⢄⣀⣀|
    +---------------+

TriangleExp(;λ0, λ1, period, γ)

Triangle wave function with exponential amplitude decay at rate γ

 Cyclic 
    +---------------+ 
    |⠀⡼⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀|
    |⢰⠁⢇⠀⡰⢇⠀⠀⡀⠀⠀⠀⠀⠀⠀|
    |⡎⠀⠘⣴⠁⠈⢆⠎⠈⢆⣀⠜⠢⣀⠤|
    +---------------+

Sin(;λ0, λ1, period)

Sine function

 Cyclic 
    +---------------+ 
    |⠀⡼⡀⠀⡸⡇⠀⢠⢣⠀⠀⡾⡀⠀⡸|
    |⢰⠁⢇⢀⠇⢸⠀⡜⠈⡆⢸⠀⢣⢀⠇|
    |⡎⠀⠘⡜⠀⠀⢷⠁⠀⢱⡇⠀⠘⡜⠀|
    +---------------+

SinDecay2(;λ0, λ1, period)

Sine function with half the amplitude every period

 Cyclic 
    +---------------+ 
    |⠀⡼⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀|
    |⢰⠁⢇⠀⢠⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀|
    |⡎⠀⠘⣤⠃⠘⣄⡠⠒⢄⣀⡠⢄⣀⣀|
    +---------------+

SinExp(;λ0, λ1, period)

Sine function with exponential amplitude decay at rate γ

 Cyclic 
    +---------------+ 
    |⠀⡼⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀|
    |⢰⠁⢇⠀⡜⢇⠀⠀⡀⠀⠀⠀⠀⠀⠀|
    |⡎⠀⠘⡼⠀⠈⢆⠎⠈⢆⣀⠜⠢⣀⠤|
    +---------------+

Cos(;λ0, λ1, period)

Cosine annealing

 Cyclic 
    +---------------+ 
    |⠉⠒⡄⠀⠀⠀⡎⠑⠢⡀⠀⠀⢠⠋⠒|
    |⠀⠀⠈⢆⠀⢰⠁⠀⠀⠑⡄⠀⡜⠀⠀|
    |⠀⠀⠀⠀⠣⡇⠀⠀⠀⠀⠈⢶⠁⠀⠀|
    +---------------+