Model Comparison with MLJFlux

This demonstration is available as a Jupyter notebook or julia script here.

In this workflow example, we see how we can compare different machine learning models with a neural network from MLJFlux.

This script tested using Julia 1.10

Basic Imports

using MLJ               # Has MLJFlux models
using Flux              # For more flexibility
using DataFrames        # To visualize hyperparameter search results
import Optimisers       # native Flux.jl optimisers no longer supported
using Measurements       # to get ± functionality
import CategoricalArrays.unwrap
using StableRNGs        # for reproducibility across Julia versions

stable_rng() = StableRNG(123)

stable_rng (generic function with 1 method)

Loading and Splitting the Data

iris = load_iris() # a named-tuple of vectors
y, X = unpack(iris, ==(:target), rng=stable_rng())

(CategoricalArrays.CategoricalValue{String, UInt32}["versicolor", "virginica", "virginica", "setosa", "virginica", "virginica", "versicolor", "setosa", "virginica", "versicolor"  …  "setosa", "virginica", "virginica", "setosa", "versicolor", "setosa", "virginica", "versicolor", "versicolor", "setosa"], (sepal_length = [6.1, 7.3, 6.3, 4.8, 5.9, 7.1, 6.7, 5.4, 6.0, 6.9  …  5.0, 6.4, 5.7, 4.6, 5.5, 4.6, 5.6, 5.7, 6.0, 5.0], sepal_width = [2.9, 2.9, 3.4, 3.4, 3.0, 3.0, 3.0, 3.9, 3.0, 3.1  …  3.3, 2.7, 2.5, 3.2, 2.4, 3.1, 2.8, 3.0, 2.9, 3.5], petal_length = [4.7, 6.3, 5.6, 1.9, 5.1, 5.9, 5.0, 1.7, 4.8, 4.9  …  1.4, 5.3, 5.0, 1.4, 3.7, 1.5, 4.9, 4.2, 4.5, 1.6], petal_width = [1.4, 1.8, 2.4, 0.2, 1.8, 2.1, 1.7, 0.4, 1.8, 1.5  …  0.2, 1.9, 2.0, 0.2, 1.0, 0.2, 2.0, 1.2, 1.5, 0.6]))

Instantiating the models Now let's construct our model. This follows a similar setup

to the one followed in the Quick Start.

NeuralNetworkClassifier = @load NeuralNetworkClassifier pkg=MLJFlux

clf1 = NeuralNetworkClassifier(
    builder=MLJFlux.MLP(; hidden=(5,4), σ=Flux.relu),
    optimiser=Optimisers.Adam(0.01),
    batch_size=8,
    epochs=50,
    rng=stable_rng(),
    )

NeuralNetworkClassifier(
  builder = MLP(
        hidden = (5, 4), 
        σ = NNlib.relu), 
  finaliser = NNlib.softmax, 
  optimiser = Adam(eta=0.01, beta=(0.9, 0.999), epsilon=1.0e-8), 
  loss = Flux.Losses.crossentropy, 
  epochs = 50, 
  batch_size = 8, 
  lambda = 0.0, 
  alpha = 0.0, 
  rng = StableRNGs.LehmerRNG(state=0x000000000000000000000000000000f7), 
  optimiser_changes_trigger_retraining = false, 
  acceleration = CPU1{Nothing}(nothing), 
  embedding_dims = Dict{Symbol, Real}())

Let's as well load and construct three other classical machine learning models:

BayesianLDA = @load BayesianLDA pkg=MultivariateStats
clf2 = BayesianLDA()
RandomForestClassifier = @load RandomForestClassifier pkg=DecisionTree
clf3 = RandomForestClassifier()
XGBoostClassifier = @load XGBoostClassifier pkg=XGBoost
clf4 = XGBoostClassifier();

[ Info: For silent loading, specify `verbosity=0`.
import MLJMultivariateStatsInterface ✔
[ Info: For silent loading, specify `verbosity=0`.
import MLJDecisionTreeInterface ✔
[ Info: For silent loading, specify `verbosity=0`.
import MLJXGBoostInterface ✔

Wrapping One of the Models in a TunedModel

Instead of just comparing with four models with the default/given hyperparameters, we will give XGBoostClassifier an unfair advantage By wrapping it in a TunedModel that considers the best learning rate η for the model.

r1 = range(clf4, :eta, lower=0.01, upper=0.5, scale=:log10)
tuned_model_xg = TunedModel(
    model=clf4,
    ranges=[r1],
    tuning=Grid(resolution=10),
    resampling=CV(nfolds=5, rng=stable_rng()),
    measure=cross_entropy,
);

Of course, one can wrap each of the four in a TunedModel if they are interested in comparing the models over a large set of their hyperparameters.

Comparing the models

We simply pass the four models to the models argument of the TunedModel construct

tuned_model = TunedModel(
    models=[clf1, clf2, clf3, tuned_model_xg],
    tuning=Explicit(),
    resampling=CV(nfolds=2, rng=stable_rng()),
    repeats=5,
    measure=cross_entropy,
);

Notice here we are using 5 x 2 Monte Carlo cross-validation.

Then wrapping our tuned model in a machine and fitting it.

mach = machine(tuned_model, X, y);
fit!(mach, verbosity=0);

Now let's see the history for more details on the performance for each of the models

history = report(mach).history
history_df = DataFrame(
    mlp = [x.model for x in history],
    measurement = [
        x.evaluation.measurement[1] ±
            x.evaluation.uncertainty_radius_95[1] for x in history
                ],
)
sort!(history_df, [order(:measurement)])

This is Occam's razor in practice.

This page was generated using Literate.jl.