How to train a model
using
FastAI
,
FastVision
,
FastMakie
,
Metalhead
import
CairoMakieFinding a learning rate
Using a good learning rate is important for a balance between model convergence and training speed, but finding one isn't always easy. FastAI.jl includes a learning rate finder that runs a mock training run with increasing learning rates to find a good one. You can use it with
lrfind.
data
,
blocks
=
load
(
datarecipes
(
)
[
"
imagenette2-160
"
]
)
task
=
ImageClassificationSingle
(
blocks
)
learner
=
tasklearner
(
task
,
data
;
callbacks
=
[
ToGPU
(
)
,
Metrics
(
accuracy
)
]
)
finderresult
=
lrfind
(
learner
)
Epoch 1 TrainingPhase(): 11%|██▉ | ETA: 0:19:29m┌───────────────┬───────┬─────────┬──────────┐
│ Phase │ Epoch │ Loss │ Accuracy │
├───────────────┼───────┼─────────┼──────────┤
│ TrainingPhase │ 1.0 │ 3.26267 │ 0.15083 │
└───────────────┴───────┴─────────┴──────────┘
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀Learning rate finder result⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
┌─────────────────────────────────────────────────────────────────┐
9 │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀0.00030206918⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣀⡄⡎│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣴⢀⠇⠘⡇│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣄⠀⢀⠜⠈⣾⠀⠀⠀│
│⢀⠤⡀⠀⠀⠀⡠⢆⢀⠎⠦⣀⢄⠀⢀⣀⠀⠀⠀⠀⠀⠀⢀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⡇⠀⠀⠀⠀⢀⠀⢀⡆⠀⠀⠀⠀⣦⡸⠀⠑⠁⠀⠀⠋⠀⠀⠀│
│⡎⠀⠉⠉⠑⠜⠀⠈⠉⠀⠀⠁⠀⠉⠉⠀⢇⠤⠒⠑⠒⠒⠁⠱⢄⠔⠑⠢⣠⡀⡜⡄⣀⡀⢀⣀⠀⡇⠀⣀⠀⡇⠀⡀⠀⢀⠿⡀⡸⠸⡀⡰⠲⣠⠃⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀│
│⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠘⠁⠈⠀⠈⠁⠀⠣⡏⠒⠁⠣⣇⡸⢣⠔⠊⠀⠓⠁⠀⠱⠃⠀⠻⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
1 │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│
└─────────────────────────────────────────────────────────────────┘
⠀10⁻⁶⸱⁹²⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀10⁻¹⠀
Estimator Suggestion
Steepest 0.000691831
MinDivByTen 0.000301995
We can also use the Makie backend to show the results of the learning rate finder:
FastMakie
.
Makie
.
plot
(
finderresult
)
Training a model from scratch
When using randomly intialized models like you can use
fitonecycle! to train:
using
Metalhead
data
,
blocks
=
load
(
datarecipes
(
)
[
"
imagenette2-160
"
]
)
task
=
ImageClassificationSingle
(
blocks
)
backbone
=
Metalhead
.
ResNet
(
18
)
.
layers
[
1
:
end
-
1
]
learner
=
tasklearner
(
task
,
data
;
callbacks
=
[
ToGPU
(
)
,
Metrics
(
accuracy
)
]
)
fitonecycle!
(
learner
,
10
,
0.001
)Finetuning a pretrained model
When finetuning a pretrained model, it is recommended to use
finetune! which uses a warmup schedule to train the newly initiliazed head more quickly than the pretrained backbone.
data
,
blocks
=
loaddataset
(
"
imagenette2-160
"
,
(
Image
,
Label
)
)
task
=
ImageClassificationSingle
(
blocks
)
learner
=
tasklearner
(
task
,
data
;
backbone
=
Metalhead
.
ResNet50
(
pretrain
=
true
)
.
layers
[
1
]
[
1
:
end
-
1
]
,
callbacks
=
[
ToGPU
(
)
,
Metrics
(
accuracy
)
]
)
finetune!
(
learner
,
5
,
0.0005
)The following pages link back here:
Saving and loading models for inference, fastai API comparison