Flux provides the DataLoader type in the Flux.Data module to handle iteration over mini-batches of data.

Flux.Data.DataLoaderType
DataLoader(data...; batchsize=1, shuffle=false, partial=true)

An object that iterates over mini-batches of data, each mini-batch containing batchsize observations (except possibly the last one).

Takes as input one or more data tensors, e.g. X in unsupervised learning, X and Y in supervised learning. The last dimension in each tensor is considered to be the observation dimension.

If shuffle=true, shuffles the observations each time iterations are re-started. If partial=false, drops the last mini-batch if it is smaller than the batchsize.

The original data is preserved as a tuple in the data field of the DataLoader.

Example usage:

Xtrain = rand(10, 100)
# iterate over 50 mini-batches of size 2
@assert size(x) == (10, 2)
...
end

Xtrain = rand(10, 100)
Ytrain = rand(100)
Flux.train!(loss, ps, ncycle(train_loader, 10), opt)