Dataset-Level Interpreters

Training Dynamics

class interpretdl.TrainingDynamics(paddle_model: callable, device: str = 'gpu:0')

Training Dynamics Interpreter focus the behavior of each training sample by running a normal SGD training process.

By recording the training dynamics, interpreter can diagnose dataset with hand-designed features or by learning solution.

After interpretation on the level of data, we can handle better the datasets with underlying label noises, thus can achieve a better performance on it.

More training dynamics based methods including [Forgetting Events] and [Dataset Mapping] will be available in this interpreter.

Parameters:

• paddle_model (callable) – A model with forward() and possibly backward() functions.
• device (str) – The device used for running paddle_model, options: "cpu", "gpu:0", "gpu:1" etc.

generator(train_loader: callable, optimizer: <module 'paddle.optimizer' from '/home/docs/checkouts/readthedocs.org/user_builds/interpretdl/envs/latest/lib/python3.7/site-packages/paddle/optimizer/__init__.py'>, epochs: int)

Run the training process and record the forgetting events statistics.

Parameters:

• train_loader (callable) – A training data generator.
• optimizer (paddle.optimizer) – The paddle optimizer.
• epochs (int) – The number of epochs to train the model.

Returns:

A pointwise training dynamics(history) for each epoch.

Return type:

training_dynamics (dict)

save(logits, assigned_targets, label_flip=None, save_path=None)

Save transformed training dynamics .

Parameters:save_path (_type_, optional) – The filepath to save the processed image. If None, the image will not be saved. Default: None

transform(logits, assigned_targets)

Transform training dynamics with linear interpolation.

Parameters:

• logits (dict) – A dictionary recording training dynamics.
• assigned_targets (list) – The assigned targets of dataset,.

Beyond Hand-designed Feature Interpreter

class interpretdl.BHDFInterpreter(detector: callable = None, device: str = 'gpu:0')

BHDFInterpreter takes the training dynamics as raw input and lets an LSTM model to predict whether the sample is mislabeled or clean. We have provided a pre-trained LSTM, which can be directly used for identifying the mislabels.

# TODO: to add the arxiv link. More details regarding this method can be found in the original paper: [link_to_be_annonced]().

For reproduction experiments, refer to [this repo](https://github.com/Christophe-Jia/mislabel-detection).

Parameters:

• detector (callable, optional) – A detector model for identifying the mislabeled samples. Defaults to None.
• device (str, optional) – The device used for running detector, options: "cpu", "gpu:0", "gpu:1" etc. Defaults to ‘gpu:0’.

interpret(training_dynamics=None, training_dynamics_path='assets/training_dynamics.npz')

Call this function to rank samples’ correctness.

Parameters:

• training_dynamics (dict, optional) – Training dynamics is a dictionary, which has keys as follows:
• { –
• label_flip – list: The position of label contamination where True indicates label noise;
• labels – numpy.ndarray: with shape of length of dataset * class number, generated by TrainingDynamics.generator;
• td – numpy.ndarray: with shape of length of dataset * training epochs * class number, point-wise probability for each epoch, generated by TrainingDynamics.generator.
• } –

Returns:

(order,predictions) where order is {ranking of label correctness in form of data indices list} and predictions is {point-wise predictions as clean}.

Return type:

(numpy.ndarray, list)

Forgetting Events

class interpretdl.ForgettingEventsInterpreter(model: callable, device: str = 'gpu:0')

Forgetting Events Interpreter computes the frequency of forgetting events for each training sample by running a normal training process. The training sample undergoes a forgetting event if it is misclassified at step t+1 after having been correctly classified at step t.

A training sample would be more probable to be mislabeled or hard to learn if it has more forgetting events happened.

More details regarding the Forgetting Events method can be found in the original paper: https://arxiv.org/abs/1812.05159.

Parameters:

• model (callable) – A model with forward() and possibly backward() functions.
• device (str) – The device used for running model, options: "cpu", "gpu:0", "gpu:1" etc.

interpret(train_reader: callable, optimizer: <module 'paddle.optimizer' from '/home/docs/checkouts/readthedocs.org/user_builds/interpretdl/envs/latest/lib/python3.7/site-packages/paddle/optimizer/__init__.py'>, batch_size: int, epochs: int, find_noisy_labels=False, save_path=None)

Run the training process and record the forgetting events statistics.

Parameters:

• train_reader (callable) – A training data generator.
• optimizer (paddle.optimizer) – The paddle optimizer.
• batch_size (int) – Number of samples to forward each time.
• epochs (int) – The number of epochs to train the model.
• find_noisy_labels (bool, optional) – whether to find noisy labels. Defaults to False.
• save_path (_type_, optional) – The filepath to save the processed image. If None, the image will not be saved. Default: None

Returns:

(count_forgotten, forgotten) where count_forgotten is {count of forgetting events: list of data indices with such count of forgetting events} and forgotten is {data index: numpy.ndarray of wrong predictions that follow true predictions in the training process}.

Return type:

(dict, dict)

SGDNoise

See Tutorials.

TrainIng Data analYzer (TIDY)

See Tutorials.