Experiment

Warning

This class is still in Beta phase. It may well be considerably modified in near future.

class poutyne.framework.Experiment(directory, module, *, device=None, logging=True, optimizer='sgd', loss_function=None, batch_metrics=None, epoch_metrics=None, monitor_metric=None, monitor_mode=None, task=None)[source]

The Experiment class provides a straightforward experimentation tool for efficient and entirely customizable finetuning of the whole neural network training procedure with PyTorch. The Experiment object takes care of the training and testing processes while also managing to keep traces of all pertinent information via the automatic logging option.

Parameters
  • directory (str) – Path to the experiment’s working directory. Will be used for the automatic logging.

  • model (torch.nn.Module) – A PyTorch module.

  • device (torch.torch.device) – The device to which the model is sent. If None, the model will be kept on its current device. (Default value = None)

  • logging (bool) – Whether or not to log the experiment’s progress. If true, various logging callbacks will be inserted to output training and testing stats as well as to automatically save model checkpoints, for example. See train() and test() for more details. (Default value = True)

  • optimizer (Union[torch.optim.Optimizer, str]) – If Pytorch Optimizer, must already be initialized. If str, should be the optimizer’s name in Pytorch (i.e. ‘Adam’ for torch.optim.Adam). (Default value = ‘sgd’)

  • loss_function (Union[Callable, str]) – loss layer or custom loss function. It can also be a string with the same name as a PyTorch loss function (either the functional or object name). The loss function must have the signature loss_function(input, target) where input is the prediction of the network and target is the ground truth. If None, will default to, in priority order, either the model’s own loss function or the default loss function associated with the task. (Default value = None)

  • batch_metrics (list) – List of functions with the same signature as the loss function. Each metric can be any PyTorch loss function. It can also be a string with the same name as a PyTorch loss function (either the functional or object name). ‘accuracy’ (or just ‘acc’) is also a valid metric. Each metric function is called on each batch of the optimization and on the validation batches at the end of the epoch. (Default value = None)

  • epoch_metrics (list) – List of functions with the same signature as EpochMetric (Default value = None)

  • monitor_metric (str) – Which metric to consider for best model performance calculation. Should be in the format ‘{metric_name}’ or ‘val_{metric_name}’ (i.e. ‘val_loss’). If None, will follow the value suggested by task or default to ‘val_loss’. (Default value = None)

  • monitor_mode (str) – Which mode, either ‘min’ or ‘max’, should be used when considering the monitor_metric value. If None, will follow the value suggested by task or default ‘min’. (Default value = None)

  • task (str) – Any str beginning with either ‘classif’ or ‘reg’. Specifying a task can assign default values to the loss_function, batch_metrics, monitor_mode and monitor_mode. For task that begins with ‘reg’, the only default value is the loss function that is the mean squared error. When beginning with ‘classif’, the default loss function is the cross-entropy loss, the default batch metrics will be the accuracy and the default monitoring will be set on ‘val_acc’ with a ‘max’ mode. (Default value = None)

Example

Using a PyTorch DataLoader, on classification task with SGD optimizer:

import torch
from torch.utils.data import DataLoader, TensorDataset
from poutyne.framework import Experiment

num_features = 20
num_classes = 5

# Our training dataset with 800 samples.
num_train_samples = 800
train_x = torch.rand(num_train_samples, num_features)
train_y = torch.randint(num_classes, (num_train_samples, ), dtype=torch.long)
train_dataset = TensorDataset(train_x, train_y)
train_generator = DataLoader(train_dataset, batch_size=32)

# Our validation dataset with 200 samples.
num_valid_samples = 200
valid_x = torch.rand(num_valid_samples, num_features)
valid_y = torch.randint(num_classes, (num_valid_samples, ), dtype=torch.long)
valid_dataset = TensorDataset(valid_x, valid_y)
valid_generator = DataLoader(valid_dataset, batch_size=32)

# Our network
pytorch_module = torch.nn.Linear(num_features, num_train_samples)

# Intialization of our experimentation and network training
exp = Experiment('./simple_example',
                 pytorch_module,
                 optimizer='sgd',
                 task='classif')
exp.train(train_generator, valid_generator, epochs=5)

The above code will yield an output similar to the below lines. Note the automatic checkpoint saving in the experiment directory when the monitored metric improved.

Epoch 1/5 0.09s Step 25/25: loss: 6.351375, acc: 1.375000, val_loss: 6.236106, val_acc: 5.000000
Epoch 1: val_acc improved from -inf to 5.00000, saving file to ./simple_example/checkpoint_epoch_1.ckpt
Epoch 2/5 0.10s Step 25/25: loss: 6.054254, acc: 14.000000, val_loss: 5.944495, val_acc: 19.500000
Epoch 2: val_acc improved from 5.00000 to 19.50000, saving file to ./simple_example/checkpoint_epoch_2.ckpt
Epoch 3/5 0.09s Step 25/25: loss: 5.759377, acc: 22.875000, val_loss: 5.655412, val_acc: 21.000000
Epoch 3: val_acc improved from 19.50000 to 21.00000, saving file to ./simple_example/checkpoint_epoch_3.ckpt
...

Training can now easily be resumed from the best checkpoint:

exp.train(train_generator, valid_generator, epochs=10)
Restoring model from ./simple_example/checkpoint_epoch_3.ckpt
Loading weights from ./simple_example/checkpoint.ckpt and starting at epoch 6.
Loading optimizer state from ./simple_example/checkpoint.optim and starting at epoch 6.
Epoch 6/10 0.16s Step 25/25: loss: 4.897135, acc: 22.875000, val_loss: 4.813141, val_acc: 20.500000
Epoch 7/10 0.10s Step 25/25: loss: 4.621514, acc: 22.625000, val_loss: 4.545359, val_acc: 20.500000
Epoch 8/10 0.24s Step 25/25: loss: 4.354721, acc: 23.625000, val_loss: 4.287117, val_acc: 20.500000
...

Testing is also very intuitive:

exp.test(test_generator)
Restoring model from ./simple_example/checkpoint_epoch_9.ckpt
Found best checkpoint at epoch: 9
lr: 0.01, loss: 4.09892, acc: 23.625, val_loss: 4.04057, val_acc: 21.5
On best model: test_loss: 4.06664, test_acc: 17.5

Finally, all the pertinent metrics specified to the Experiment at each epoch are stored in a specific logging file, found here at ‘./simple_example/log.tsv’.

epoch       time                lr      loss                    acc     val_loss            val_acc
1       0.0721172170015052  0.01    6.351375141143799       1.375   6.23610631942749        5.0
2       0.0298177790245972  0.01    6.054253826141357       14.000  5.94449516296386        19.5
3       0.0637106419890187  0.01    5.759376544952392       22.875  5.65541223526001        21.0
...
get_best_epoch_stats()[source]

Returns all computed statistics corresponding to the best epoch according to the monitor_metric and monitor_mode attributes.

Returns

dict where each key is a column name in the logging output file and values are the ones found at the best epoch.

load_checkpoint(checkpoint, *, verbose=False)[source]

Loads the attribute model’s weights with the weights at a given checkpoint epoch.

Parameters
  • checkpoint (Union[int, str]) – Which checkpoint to load the model’s weights form. If ‘best’, will load the best weights according to monitor_metric and monitor_mode. If ‘last’, will load the last model checkpoint. If int, will load the checkpoint of the specified epoch.

  • verbose (bool, optional) – Whether or not to print the best epoch number and stats when checkpoint is ‘best’. (Default value = False)

Returns

If checkpoint is ‘best’, will return the best epoch stats, as per get_best_epoch_stats(), else None.

test(test_generator, *, steps=None, checkpoint='best', seed=42)[source]

Computes and returns the loss and the metrics of the attribute model on a given test examples generator.

If the Experiment has logging enabled (i.e. self.logging is True), test and validation statistics are saved in a specific test output .tsv file.

Parameters
  • test_generator – Generator-like object for the test set. See fit_generator() for details on the types of generators supported.

  • steps (int, optional) – Number of iterations done on generator. (Defaults the number of steps needed to see the entire dataset)

  • checkpoint (Union[str, int]) – Which model checkpoint weights to load for the test evaluation. If ‘best’, will load the best weights according to monitor_metric and monitor_mode. If ‘last’, will load the last model checkpoint. If int, will load the checkpoint of the specified epoch. (Default value = ‘best’)

  • seed (int, optional) – Seed used to make the sampling deterministic. (Default value = 42)

Returns

dict sorting of all the test metrics values by their names.

train(train_generator, valid_generator=None, *, callbacks=None, lr_schedulers=None, save_every_epoch=False, disable_tensorboard=False, epochs=1000, steps_per_epoch=None, validation_steps=None, batches_per_step=1, seed=42)[source]

Trains or finetunes the attribute model on a dataset using a generator. If a previous training already occured and lasted a total of n_previous epochs, then the model’s weights will be set to the last checkpoint and the training will be resumed for epochs range (n_previous, epochs].

If the Experiment has logging enabled (i.e. self.logging is True), numerous callbacks will be automatically included. Notably, two ModelCheckpoint objects will take care of saving the last and every new best (according to monitor mode) model weights in appropriate checkpoint files. OptimizerCheckpoint and LRSchedulerCheckpoint will also respectively handle the saving of the optimizer and LR scheduler’s respective states for future retrieval. Moreover, a CSVLogger will save all available epoch statistics in an output .tsv file. Lastly, a TensorBoardLogger handles automatic TensorBoard logging of various neural network statistics.

Parameters
  • train_generator – Generator-like object for the training set. See fit_generator() for details on the types of generators supported.

  • valid_generator (optional) – Generator-like object for the validation set. See fit_generator() for details on the types of generators supported. (Default value = None)

  • callbacks (List[Callback]) – List of callbacks that will be called during training. (Default value = None)

  • lr_schedulers (List[_PyTorchLRSchedulerWrapper]) – List of learning rate schedulers. (Default value = None)

  • save_every_epoch (bool, optional) – Whether or not to save the experiment model’s weights after every epoch. (Default value = False)

  • disable_tensorboard (bool, optional) – Wheter or not to disable the automatic tensorboard logging callbacks. (Default value = False)

  • epochs (int) – Number of times the entire training dataset is seen. (Default value = 1000)

  • steps_per_epoch (int, optional) – Number of batch used during one epoch. Obviously, using this argument may cause one epoch not to see the entire training dataset or see it multiple times. (Defaults the number of steps needed to see the entire training dataset)

  • validation_steps (int, optional) – Same as for steps_per_epoch but for the validation dataset. (Defaults to steps_per_epoch if provided or the number of steps needed to see the entire validation dataset)

  • batches_per_step (int) – Number of batches on which to compute the running loss before backpropagating it through the network. Note that the total loss used for backpropagation is the mean of the batches_per_step batch losses. (Default value = 1)

  • seed (int, optional) – Seed used to make the sampling deterministic. (Default value = 42)

Returns

List of dict containing the history of each epoch.