Input Feature Based Interpreters

Consensus

class interpretdl.ConsensusInterpreter(InterpreterClass, list_of_models: list, device: str = 'gpu:0', **kwargs)

ConsensusInterpreter averages the explanations of a given Interpreter over a list of models. The averaged result is more like an explanation for the data, instead of specific models. For visual object recognition tasks, the Consensus explanation would be more aligned with the object than individual models.

More details regarding the Consensus method can be found in the original paper: https://arxiv.org/abs/2109.00707.

For reference, the list_of_models can be found from paddle.vision.models or PPClas.

Parameters:

• InterpreterClass ([type]) – The given Interpreter defined in InterpretDL.
• list_of_models (list) – a list of trained models.
• device (str) – The device used for running model, options: "cpu", "gpu:0", "gpu:1" etc.

interpret(inputs: str, **kwargs) → numpy.ndarray

The technical details are simple to understand for the Consensus method: Given the inputs and the interpretation algorithm (one of the Interpreters), each model in list_of_models will produce an explanation, then Consensus will concatenate all the explanations. Subsequent normalization and average can be done as users’ preference. The suggested operation for input gradient based algorithms is average of the absolute values.

We leave the visualization to users. See the notebook example for an example.

import interpretdl as it
from paddle.vision.models import resnet34, resnet50, resnet101, mobilenet_v2

list_models = {
    'resnet34': resnet34(pretrained=True),
    'resnet50': resnet50(pretrained=True),
    'resnet101': resnet101(pretrained=True),
    'mobilenet_v2': mobilenet_v2(pretrained=True)
}
consensus = ConsensusInterpreter(it.SmoothGradInterpreter, list_models.values(), device='gpu:0')

import matplotlib.pyplot as plt
import numpy as np

cols = len(list_models) + 1
psize = 4
fig, ax = plt.subplots(1, cols, figsize=(cols*psize, 1*psize))

for axis in ax:
    axis.axis('off')

for i in range(len(list_models)):
    ax[i].imshow(np.abs(exp[i]).sum(0))
    ax[i].set_title(list(list_models.keys())[i])

ax[-1].imshow(np.abs(exp).sum(1).mean(0))
ax[-1].set_title('Consensus')

Parameters:inputs (str or list of strs or numpy.ndarray) – The input image filepath or a list of filepaths or numpy array of read images. Returns:Concatenated raw explanations. Return type:np.ndarray

Gradient Shap

class interpretdl.GradShapCVInterpreter(model: callable, device: str = 'gpu:0')

Gradient SHAP Interpreter for CV tasks.

For input gradient based interpreters, the target issue is generally the vanilla input gradient’s noises. The basic idea of reducing the noises is to use different similar inputs to get the input gradients and do the average.

GradShap uses noised inputs to get input gradients and then average.

More details regarding the GradShap method can be found in the original paper: http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.

Parameters:

• model (callable) – A model with forward() and possibly backward() functions.
• device (str) – The device used for running model, options: "cpu", "gpu:0", "gpu:1" etc.

interpret(inputs: str, labels: list = None, baselines: numpy.ndarray = None, n_samples: int = 5, noise_amount: float = 0.1, gradient_of: str = 'probability', resize_to: int = 224, crop_to: int = None, visual: bool = True, save_path: str = None) → numpy.ndarray

The technical details of the GradShap method are described as follows: GradShap generates n_samples noised inputs, with the noise scale of noise_amount, and then computes the gradients w.r.t. these noised inputs. A difference between baselines and noised inputs is considered. The final explanation is the multiplication between the gradients and the difference to baselines.

Parameters:

• inputs (str or list) – The input image filepath or a list of filepaths or numpy array of read images.
• labels (list or np.ndarray, optional) – The target labels to analyze. The number of labels should be equal to the number of images. If None, the most likely label for each image will be used. Default: None.
• baselines (np.ndarray, optional) – The baseline images to compare with. It should have the same shape as images and same length as the number of images. If None, the baselines of all zeros will be used. Default: None.
• n_samples (int, optional) – The number of randomly generated samples. Defaults to 5.
• noise_amount (float, optional) – Noise level of added noise to each image. The std of Gaussian random noise is noise_amount * (x _max - x _min). Default: 0.1.
• gradient_of (str, optional) – compute the gradient of [‘probability’, ‘logit’ or ‘loss’]. Default: 'probability'. SmoothGrad uses probability for all tasks by default.
• resize_to (int, optional) – Images will be rescaled with the shorter edge being resize_to. Defaults to 224.
• crop_to (int, optional) – After resize, images will be center cropped to a square image with the size crop_to. If None, no crop will be performed. Defaults to None.
• visual (bool, optional) – Whether or not to visualize the processed image. Default: True.
• save_path (str, optional) – The filepath(s) to save the processed image(s). If None, the image will not be saved. Default: None.

Returns:

the explanation result.

Return type:

np.ndarray

class interpretdl.GradShapNLPInterpreter(model: callable, device: str = 'gpu:0')

TODO: Inherit from a subabstract interpreter. Gradient SHAP Interpreter for NLP tasks.

For input gradient based interpreters, the target issue is generally the vanilla input gradient’s noises. The basic idea of reducing the noises is to use different similar inputs to get the input gradients and do the average.

The inputs for NLP tasks are considered as the embedding features. So the noises or the changes of inputs are done for the embeddings.

More details regarding the GradShap method can be found in the original paper: http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.

Parameters:

• model (callable) – A model with forward() and possibly backward() functions.
• device (str) – The device used for running model, options: "cpu", "gpu:0", "gpu:1" etc.

interpret(data: tuple, labels: list = None, n_samples: int = 5, noise_amount: float = 0.1, embedding_name: str = 'word_embeddings', return_pred: bool = True) → numpy.ndarray

The technical details of the GradShap method for NLP tasks are similar for CV tasks, except the noises are added on the embeddings.

Parameters:

• data (tupleornp.ndarray) – The inputs to the NLP model.
• labels (listornp.ndarray, optional) – The target label to analyze. If None, the most likely label will be used. Default: None.
• n_samples (int, optional) – The number of randomly generated samples. Defaults to 5.
• noise_amount (float, optional) – Noise level of added noise to the embeddings. The std of Gaussian random noise is noise_amount * embedding.mean() * (x _max - x _min). Default: 0.1.
• embedding_name (str, optional) – name of the embedding layer at which the noises will be applied. The name of embedding can be verified through print(model). Defaults to word_embeddings.
• return_pred (bool, optional) – Whether or not to return predicted labels and probabilities. If True, a tuple of predicted labels, probabilities, and interpretations will be returned. There are useful for visualization. Else, only interpretations will be returned. Default: True.

Returns:

explanations, or (explanations, pred).

Return type:

np.ndarray or tuple

Integrated Gradients

class interpretdl.IntGradCVInterpreter(model: callable, device: str = 'gpu:0', **kwargs)

Integrated Gradients Interpreter for CV tasks.

For input gradient based interpreters, the target issue is generally the vanilla input gradient’s noises. The basic idea of reducing the noises is to use different similar inputs to get the input gradients and do the average.

IntGrad uses the Riemann approximation of the integral, i.e., interpolated values between a baseline (zero) and the original input as inputs, and computes the gradients which will be averaged as the final explanation.

More details regarding the Integrated Gradients method can be found in the original paper: https://arxiv.org/abs/1703.01365.

Parameters:

• model (callable) – A model with forward() and possibly backward() functions.
• device (str) – The device used for running model, options: "cpu", "gpu:0", "gpu:1" etc.

interpret(inputs: str, labels: list = None, baselines: numpy.ndarray = None, steps: int = 50, num_random_trials: int = 10, gradient_of: str = 'probability', resize_to: int = 224, crop_to: int = None, visual: bool = True, save_path: str = None) → numpy.ndarray

The technical details of the IntGrad method are described as follows: Given inputs, IntGrad interpolates steps points between baselines (usually set to zeros) and inputs. baselines can be set to random, so that num_random_trials baselines are used, instead of zeros. Then IntGrad computes the gradients w.r.t. these interpolated values and averages the results as final explanation.

Parameters:

• inputs (str or list) – The input image filepath or a list of filepaths or numpy array of read images.
• labels (list or tuple or np.ndarray or None, optional) – The target labels to analyze. The number of labels should be equal to the number of images. If None, the most likely label for each image will be used. Default: None.
• baselines (np.ndarray or None, optional) – The baseline images to compare with. It should have the same shape as images and same length as the number of images. If None, the baselines of all zeros will be used. Default: None.
• steps (int, optional) – number of steps in the Riemann approximation of the integral. Default: 50.
• num_random_trials (int, optional) – number of random initializations to take average in the end. Default: 10.
• gradient_of (str, optional) – compute the gradient of [‘probability’, ‘logit’ or ‘loss’]. Default: 'probability'. Multi-class classification uses probabitliy, while binary classification uses logit.
• resize_to (int, optional) – Images will be rescaled with the shorter edge being resize_to. Defaults to 224.
• crop_to (int, optional) – After resize, images will be center cropped to a square image with the size crop_to. If None, no crop will be performed. Defaults to None.
• visual (bool, optional) – Whether or not to visualize the processed image. Default: True.
• save_path (str, optional) – The filepath(s) to save the processed image(s). If None, the image will not be saved. Default: None.

Returns:

the explanation result.

Return type:

np.ndarray

class interpretdl.IntGradNLPInterpreter(model: callable, device: str = 'gpu:0', **kwargs)

Integrated Gradients Interpreter for NLP tasks.

For input gradient based interpreters, the target issue is generally the vanilla input gradient’s noises. The basic idea of reducing the noises is to use different similar inputs to get the input gradients and do the average.

The inputs for NLP tasks are considered as the embedding features. So the noises or the changes of inputs are done for the embeddings.

More details regarding the Integrated Gradients method can be found in the original paper: https://arxiv.org/abs/1703.01365.

Parameters:

• model (callable) – A model with forward() and possibly backward() functions.
• device (str) – The device used for running model, options: "cpu", "gpu:0", "gpu:1" etc.

interpret(raw_text: str, tokenizer: callable = None, text_to_input_fn: callable = None, label: list = None, steps: int = 50, gradient_of: str = 'logit', embedding_name: str = 'word_embeddings', max_seq_len: int = 128, visual: bool = False) → numpy.ndarray

The technical details of the IntGrad method for NLP tasks are similar for CV tasks, except the noises are added on the embeddings.

Parameters:

• data (tupleornp.ndarray) – The inputs to the NLP model.
• labels (listornp.ndarray, optional) – The target labels to analyze. If None, the most likely label will be used. Default: None.
• steps (int, optional) – number of steps in the Riemann approximation of the integral. Default: 50.
• gradient_of (str, optional) – compute the gradient of [‘probability’, ‘logit’ or ‘loss’]. Default: 'logit'. Multi-class classification uses probabitliy, while binary classification uses logit.
• embedding_name (str, optional) – name of the embedding layer at which the noises will be applied. The name of embedding can be verified through print(model). Defaults to word_embeddings.

Returns:

explanations, or (explanations, pred).

Return type:

np.ndarray or tuple

LIME

class interpretdl.LIMECVInterpreter(model: callable, device: str = 'gpu:0', random_seed: int = None)

LIME presents a locally explanation by fitting a set of perturbed samples near the target sample using an interpretable model, specifically a linear model.

The implementation is based on https://github.com/marcotcr/lime.

More details regarding the LIME method can be found in the original paper: https://arxiv.org/abs/1602.04938.

Parameters:

• model (callable) – A model with forward() and possibly backward() functions.
• device (str) – The device used for running model, options: "cpu", "gpu:0", "gpu:1" etc.

interpret(data: str, interpret_class: int = None, top_k: int = 1, num_samples: int = 1000, batch_size: int = 50, resize_to: int = 224, crop_to: int = None, visual: bool = True, save_path: str = None)

Main function of the interpreter.

The implementation is based on https://github.com/marcotcr/lime.

Parameters:

• data (str) – The input file path.
• interpret_class (int, optional) – The index of class to interpret. If None, the most likely label will be used. Default: None.
• top_k (int, optional) – Number of top classes to interpret. Will not be used if interpret_class is given. Default: 1.
• num_samples (int, optional) – LIME sampling numbers. Larger number of samples usually gives more accurate interpretation. Default: 1000.
• batch_size (int, optional) – Number of samples to forward each time. Default: 50.
• resize_to (int, optional) – Images will be rescaled with the shorter edge being resize_to. Defaults to 224.
• crop_to (int, optional) – After resize, images will be center cropped to a square image with the size crop_to. If None, no crop will be performed. Defaults to None.
• visual (bool, optional) – Whether or not to visualize the processed image. Default: True.
• save_path (str, optional) – The filepath(s) to save the processed image(s). If None, the image will not be saved. Default: None.

Returns:

LIME results: {interpret_label_i: weights on features}

Return type:

[dict]

class interpretdl.LIMENLPInterpreter(model: callable, device: str = 'gpu:0', random_seed: int = None)

LIME Interpreter for NLP tasks.

LIME presents a locally explanation by fitting a set of perturbed samples near the target sample using an interpretable model, specifically a linear model.

The implementation is based on https://github.com/marcotcr/lime.

More details regarding the LIME method can be found in the original paper: https://arxiv.org/abs/1602.04938.

Parameters:

• model (callable) – A model with forward() and possibly backward() functions.
• device (str) – The device used for running model, options: "cpu", "gpu:0", "gpu:1" etc.
• random_seed (int) – random seed. Defaults to None.

interpret(raw_text: str, tokenizer: callable = None, text_to_input_fn: callable = None, preprocess_fn: callable = None, unk_id: int = 0, pad_id: int = 0, classes_to_interpret: list = None, num_samples: int = 1000, batch_size: int = 50, max_seq_len: int = 128, visual: bool = False)

Main function of the interpreter.

The implementation is based on https://github.com/marcotcr/lime.

Parameters:

• data (str) – The raw string for analysis.
• tokenizer (callable) –
• text_to_input (callable) – A user-defined function that convert raw text string to a tuple of inputs that can be fed into the NLP model.
• unk_id (int) – The word id to replace occluded words. Typical choices include “”, <unk>, and <pad>.
• pad_id (int or None) – The word id used to pad the sequences. If None, it means there is no padding. Default: None.
• classes_to_interpret (list or numpy.ndarray, optional) – The index of class to interpret. If None, the most likely label will be used. can be Default: None.
• num_samples (int, optional) – LIME sampling numbers. Larger number of samples usually gives more accurate interpretation. Default: 1000.
• batch_size (int, optional) – Number of samples to forward each time. Default: 50.
• visual (bool, optional) – Whether or not to visualize. Default: True.

Returns:

LIME results: {interpret_label_i: weights on features}

Return type:

[dict]

GLIME

class interpretdl.GLIMECVInterpreter(model: callable, device: str = 'gpu:0')

G-LIME CV Interpreter. This method integrates the global information from NormLIME or Average to the local explanation LIME.

More details can be found in this [pdf link](https://github.com/PaddlePaddle/InterpretDL/files/10110787/glime-aij-paper.pdf).

Parameters:

• model (callable) – A model with forward() and possibly backward() functions.
• device (str) – The device used for running model, options: "cpu", "gpu:0", "gpu:1" etc.

compute_global_weights(g_name: str = 'normlime', list_of_lime_explanations: list = None, list_file_paths: list = None, save_path: str = None)

Compute the global weights, given the list_of_lime_explanations. This is done by NormLIME or Average Global Explanations, which are introduced in https://arxiv.org/abs/1909.04200 and https://arxiv.org/abs/1907.03039 respectively.

Parameters:

• g_name (str, optional) – The method to aggregate local explanations. Defaults to 'normlime'.
• list_of_lime_explanations (list, optional) – The LIME results. Defaults to None.
• list_file_paths (list, optional) – This is not implemented currently. Defaults to None.
• save_path (str, optional) – A path to save the global weights, which can be directly used the next time, and called by set_global_weights(). Defaults to None.

Raises:

NotImplementedError – NotImplementedError.

Returns:

Global Weights.

Return type:

dict

interpret(data: str, interpret_class: int = None, top_k: int = 1, prior_method: str = 'none', prior_reg_force: float = 1.0, num_samples: int = 1000, batch_size: int = 50, resize_to: int = 224, crop_to: int = None, visual: bool = True, save_path: str = None)

Note that for GLIME interpreter, set_global_weights() needs to be called before calling interpret(). Basically, the technical process of GLIME is similar to LIME. See the tutorial for more details.

Parameters:

• data (str) – The input file path.
• interpret_class (int, optional) – The index of class to interpret. If None, the most likely label will be used. Default: None.
• top_k (int, optional) – Number of top classes to interpret. Will not be used if interpret_class is given. Default: 1.
• prior_method – Prior method. Can be chosen from {"none", "ridge"}. Defaults to "none", which is equivalent to LIME. If none, interpret() will use zeros as prior; Otherwise, the loaded prior will be used.
• prior_reg_force (float, optional) – The regularization force to apply. Default: 1.0.
• num_samples (int, optional) – LIME sampling numbers. Larger number of samples usually gives more accurate interpretation. Default: 1000.
• batch_size (int, optional) – Number of samples to forward each time. Default: 50
• resize_to (int, optional) – Images will be rescaled with the shorter edge being resize_to. Defaults to 224.
• crop_to ([type], optional) – After resize, images will be center cropped to a square image with the size crop_to. If None, no crop will be performed. Defaults to None.
• visual (bool, optional) – Whether or not to visualize the processed image. Default: True.
• save_path (str, optional) – The path to save the processed image. If None, the image will not be saved. Default: None.

Returns:

LIME results: {interpret_label_i: weights on features}

Return type:

[dict]

set_global_weights(global_weights_info: str)

Set directly the global weights without any pre-computations.

Parameters:global_weights_info (str or dict) – A path of the file or the dict.

LIME With Global Prior

LRP

class interpretdl.LRPCVInterpreter(model: callable, device: str = 'gpu:0')

Layer-wise Relevance Propagation (LRP) Interpreter for CV tasks.

The detailed introduction of LRP can be found in the tutorial. Layer-wise Relevance Propagation (LRP) is an explanation technique applicable to models structured as neural networks, where inputs can be e.g. images, videos, or text. LRP operates by propagating the prediction backwards in the neural network, by means of purposely designed local propagation rules.

Note that LRP requires model have relprop() and related implementations, see tutorial/assets/lrp_model. This is different from other interpreters, which do not have additional requirements for model.

More details regarding the LRP method can be found in the original paper: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0130140.

Parameters:

• model (callable) – A model with forward() and possibly backward() functions.
• device (str) – The device used for running model, options: "cpu", "gpu:0", "gpu:1" etc.

interpret(inputs, label=None, resize_to=224, crop_to=None, visual=True, save_path=None)

The difficulty for LRP implementation does not reside the algorithm, but the model. The model should be implemented with relprop() functions, and the algorithm calls the relevance back-propagation, until the input layer, where the final explanation is obtained.

Parameters:

• inputs (str or list of strs or numpy.ndarray) – The input image filepath or a list of filepaths or numpy array of read images.
• labels (list or tuple or numpy.ndarray, optional) – The target labels to analyze. The number of labels should be equal to the number of images. If None, the most likely label for each image will be used. Default: None.
• resize_to (int, optional) – Images will be rescaled with the shorter edge being resize_to. Defaults to 224.
• crop_to (int, optional) – After resize, images will be center cropped to a square image with the size crop_to. If None, no crop will be performed. Defaults to None.
• visual (bool, optional) – Whether or not to visualize the processed image. Default: True.
• save_path (str, optional) – The filepath(s) to save the processed image(s). If None, the image will not be saved. Default: None.

Returns:

interpretations/Relevance map for images.

Return type:

[numpy.ndarray]

Occlusion

class interpretdl.OcclusionInterpreter(model: callable, device: str = 'gpu:0')

Occlusion Interpreter.

OcclusionInterpreter follows the simple idea of perturbation that says if the most important input features are perturbed, then the model’s prediction will change the most. OcclusionInterpreter masks a block of pixels in the image, and then computes the prediction changes. According to the changes, the final explanation is obtained.

More details regarding the Occlusion method can be found in the original paper: https://arxiv.org/abs/1311.2901

Part of the code is modified from https://github.com/pytorch/captum/blob/master/captum/attr/_core/occlusion.py.

Parameters:

• model (callable) – A model with forward() and possibly backward() functions.
• device (str) – The device used for running model, options: "cpu", "gpu:0", "gpu:1" etc.

interpret(inputs: str, sliding_window_shapes: tuple, labels: int = None, strides: int = 1, baselines: numpy.ndarray = None, perturbations_per_eval: int = 1, resize_to: int = 224, crop_to: int = None, visual: bool = True, save_path: str = None)

Part of the code is modified from https://github.com/pytorch/captum/blob/master/captum/attr/_core/occlusion.py.

Parameters:

• inputs (str or list of strs or numpy.ndarray) – The input image filepath or a list of filepaths or numpy array of read images.
• sliding_window_shapes (tuple) – Shape of sliding windows to occlude data.
• labels (list or tuple or numpy.ndarray, optional) – The target labels to analyze. The number of labels should be equal to the number of images. If None, the most likely label for each image will be used. Default: None
• strides (int or tuple) – The step by which the occlusion should be shifted by in each direction for each iteration. If int, the step size in each direction will be the same. Default: 1.
• baselines (numpy.ndarray or None, optional) – The baseline images to compare with. It should have the same shape as images. If None, the baselines of all zeros will be used. Default: None.
• perturbations_per_eval (int, optional) – number of occlusions in each batch. Default: 1.
• resize_to (int, optional) – Images will be rescaled with the shorter edge being resize_to. Defaults to 224.
• crop_to (int, optional) – After resize, images will be center cropped to a square image with the size crop_to. If None, no crop will be performed. Defaults to None.
• visual (bool, optional) – Whether or not to visualize the processed image. Default: True.
• save_path (str, optional) – The filepath(s) to save the processed image(s). If None, the image will not be saved. Default: None.

Returns:

interpretations for images

Return type:

[numpy.ndarray]

Smooth Gradients

class interpretdl.SmoothGradInterpreter(model: callable, device: str = 'gpu:0', **kwargs)

Smooth Gradients Interpreter.

For input gradient based interpreters, the target issue is generally the vanilla input gradient’s noises. The basic idea of reducing the noises is to use different similar inputs to get the input gradients and do the average.

Smooth Gradients method solves the problem of meaningless local variations in partial derivatives by adding random noise to the inputs multiple times and take the average of the gradients.

More details regarding the Smooth Gradients method can be found in the original paper: http://arxiv.org/abs/1706.03825.

Parameters:

• model (callable) – A model with forward() and possibly backward() functions.
• device (str) – The device used for running model, options: "cpu", "gpu:0", "gpu:1" etc.

interpret(inputs: str, labels: list = None, noise_amount: int = 0.1, n_samples: int = 50, gradient_of: str = 'probability', resize_to: int = 224, crop_to: int = None, visual: bool = True, save_path: str = None) → numpy.ndarray

The technical details of the SmoothGrad method are described as follows: SmoothGrad generates n_samples noised inputs, with the noise scale of noise_amount, and then computes the gradients w.r.t. these noised inputs. The final explanation is averaged gradients.

Parameters:

• inputs (str or list) – The input image filepath or a list of filepaths or numpy array of read images.
• labels (list or np.ndarray, optional) – The target labels to analyze. The number of labels should be equal to the number of images. If None, the most likely label for each image will be used. Default: None.
• noise_amount (int, optional) – Noise level of added noise to the image. The std of Gaussian random noise is noise_amount * (x _max - x _min). Default: 0.1.
• n_samples (int, optional) – The number of new images generated by adding noise. Default: 50.
• gradient_of (str, optional) – compute the gradient of [‘probability’, ‘logit’ or ‘loss’]. Default: 'probability'. SmoothGrad uses probability for all tasks by default.
• resize_to (int, optional) – Images will be rescaled with the shorter edge being resize_to. Defaults to 224.
• crop_to (int, optional) – After resize, images will be center cropped to a square image with the size crop_to. If None, no crop will be performed. Defaults to None.
• visual (bool, optional) – Whether or not to visualize the processed image. Default: True.
• save_path (str, optional) – The filepath(s) to save the processed image(s). If None, the image will not be saved. Default: None.

Returns:

the explanation result.

Return type:

np.ndarray

Smooth Gradients V2

class interpretdl.SmoothGradInterpreterV2(model: callable, device: str = 'gpu:0')

Smooth Gradients Interpreter.

For input gradient based interpreters, the target issue is generally the vanilla input gradient’s noises. The basic idea of reducing the noises is to use different similar inputs to get the input gradients and do the average.

Smooth Gradients method solves the problem of meaningless local variations in partial derivatives by adding random noise to the inputs multiple times and take the average of the gradients.

This SmoothGradInterpreterV2 only optimizes the GPU usage issue where large GPU memory usage may cause an error for large models and large batch sizes.

More details regarding the Smooth Gradients method can be found in the original paper: http://arxiv.org/abs/1706.03825.

Parameters:

• model (callable) – A model with forward() and possibly backward() functions.
• device (str) – The device used for running model, options: "cpu", "gpu:0", "gpu:1" etc.

interpret(inputs: str, labels: list = None, noise_amount: int = 0.1, n_samples: int = 50, split: int = 2, gradient_of: str = 'probability', resize_to: int = 224, crop_to: int = None, visual: bool = True, save_path: str = None) → numpy.ndarray

The technical details of the SmoothGrad method are described as follows: SmoothGrad generates n_samples noised inputs, with the noise scale of noise_amount, and then computes the gradients w.r.t. these noised inputs. The final explanation is averaged gradients. The difference to SmoothGradInterpreter is an additional argument split, where total samples are divided into split parts to pass the model, to avoid large GPU memory usages.

Parameters:

• inputs (str or list) – The input image filepath or a list of filepaths or numpy array of read images.
• labels (list or np.ndarray, optional) – The target labels to analyze. The number of labels should be equal to the number of images. If None, the most likely label for each image will be used. Default: None.
• noise_amount (int, optional) – Noise level of added noise to the image. The std of Gaussian random noise is noise_amount * (x _max - x _min). Default: 0.1.
• n_samples (int, optional) – The number of new images generated by adding noise. Default: 50.
• split (int, optional) – The number of splits. Default: 2.
• gradient_of (str, optional) – compute the gradient of [‘probability’, ‘logit’ or ‘loss’]. Default: 'probability'. SmoothGrad uses probability for all tasks by default.
• resize_to (int, optional) – Images will be rescaled with the shorter edge being resize_to. Defaults to 224.
• crop_to (int, optional) – After resize, images will be center cropped to a square image with the size crop_to. If None, no crop will be performed. Defaults to None.
• visual (bool, optional) – Whether or not to visualize the processed image. Default: True.
• save_path (str, optional) – The filepath(s) to save the processed image(s). If None, the image will not be saved. Default: None.

Returns:

the explanation result.

Return type:

np.ndarray

NormLIME

class interpretdl.NormLIMECVInterpreter(model: callable, device: str = 'gpu:0')

NormLIME Interpreter for CV tasks.

(TODO) Some technical details will be complete soon.

More details regarding the NormLIME method can be found in the original paper: https://arxiv.org/abs/1909.04200.

Parameters:

• model (callable) – A model with forward() and possibly backward() functions.
• device (str) – The device used for running model, options: "cpu", "gpu:0", "gpu:1" etc.

interpret(image_paths, num_samples=1000, batch_size=50, save_path='normlime_weights.npy', temp_data_file='all_lime_weights.npz')

Main function of the interpreter.

(TODO) Some technical details will be complete soon.

Parameters:

• image_paths (list of strs) – A list of image filepaths.
• num_samples (int, optional) – LIME sampling numbers. Larger number of samples usually gives more accurate interpretation. Default: 1000
• batch_size (int, optional) – Number of samples to forward each time. Default: 50
• save_path (str, optional) – The .npy path to save the normlime weights. It is a dictionary where the key is label and value is segmentation ids with their importance. Default: ‘normlime_weights.npy’
• temp_data_file (str, optional) – The path to save the intermediate lime weights to avoid repeating computations. Default: ‘all_lime_weights.npz’. Set to None will not save the intermediate lime weights.

Returns:

Global feature importance as a dict {label_i: weights on features}

Return type:

[dict] NormLIME weights

class interpretdl.NormLIMENLPInterpreter(model: callable, device: str = 'gpu:0')

NormLIME Interpreter for NLP tasks.

More details regarding the NormLIME method can be found in the original paper: https://arxiv.org/abs/1909.04200.

Parameters:

• model (callable) – A model with forward() and possibly backward() functions.
• device (str) – The device used for running model, options: "cpu", "gpu:0", "gpu:1" etc.

interpret(list_of_raw_text, preprocess_fn: callable, num_samples: int, batch_size: int, unk_id: int = 0, pad_id: int = 0, lod_levels: int = None, save_path: str = 'normlime_weights.npy', temp_data_file: str = 'all_lime_weights.npz')

NormLIMENLPInterpreter computes the LIME results of each sample of data, normalizes and averages the LIME results. preprocess_fn is used for coping with texts, see the tutorials for an example. num_samples and batch_size are LIME arguments, for the generated samples and the batch size of each pass.

Parameters:

• list_of_raw_text (str) – The raw string for analysis.
• preprocess_fn (Callable) – A user-defined function that input raw string and outputs the a tuple of inputs to feed into the NLP model.
• num_samples (int, optional) – LIME sampling numbers. Larger number of samples usually gives more accurate interpretation. Default: 1000
• batch_size (int, optional) – Number of samples to forward each time. Default: 50
• unk_id (int) – The word id to replace occluded words. Typical choices include “”, <unk>, and <pad>.
• pad_id (int or None) – The word id used to pad the sequences. If None, it means there is no padding. Default: None.
• lod_levels (list or tuple or numpy.ndarray or None, optional) – The lod levels for model inputs. It should have the length equal to number of outputs given by preprocess_fn. If None, lod levels are all zeros. Default: None.
• save_path (str, optional) – The .npy path to save the normlime weights. It is a dictionary where the key is label and value is segmentation ids with their importance. Default: ‘normlime_weights.npy’
• temp_data_file (str, optional) – The .npz path to save the temporal LIME results, to avoid repeating the computations. Default: ‘all_lime_weights.npz’

Returns:

{label_i: weights on features}

Return type:

[dict] NormLIME weights

TAM

class interpretdl.TAMInterpreter(model: callable, device: str = 'gpu:0')

TODO: Inherit from a subabstract interpreter. Transition Attention Maps Interpreter.

This is a specific interpreter for Transformers models. TAMInterpreter assumes that the information flowing inside the Transformer model follows the Markov Chain. Within this supposition, TAMInterpreter considers the attention maps as transition matrices and computes the explanation by multiplying the initial state with the attention maps, with integrated gradients.

More details regarding the Transition_Attention_Maps method can be found in the original paper: https://openreview.net/forum?id=TT-cf6QSDaQ.

Parameters:

• model (callable) – A model with forward() and possibly backward() functions.
• device (str) – The device used for running model, options: "cpu", "gpu:0", "gpu:1" etc.

interpret(inputs: str, start_layer: int = 4, steps: int = 20, label: int = None, resize_to: int = 224, crop_to: int = None, visual: bool = True, save_path: str = None)

Given inputs, TAMInterpreter obtains all attention maps (of layers whose name matches attention_layer_pattern) and calculates their matrix multiplication. The start_layer controls the number of involved layers. The order of involving attention maps (from last layer to the first) is different from Rollout (from first to last). Then, an integrated gradients with steps is computed and multiplied to the attention result.

Parameters:

• inputs (str or list of strs or numpy.ndarray) – The input image filepath or a list of filepaths or numpy array of read images.
• start_layer (int, optional) – Compute the state from the start layer. Default: 4.
• steps (int, optional) – number of steps in the Riemann approximation of the integral. Default: 50.
• labels (list or tuple or numpy.ndarray, optional) – The target labels to analyze. The number of labels should be equal to the number of images. If None, the most likely label for each image will be used. Default: None.
• resize_to (int, optional) – Images will be rescaled with the shorter edge being resize_to. Defaults to 224.
• crop_to (int, optional) – After resize, images will be center cropped to a square image with the size crop_to. If None, no crop will be performed. Defaults to None.
• visual (bool, optional) – Whether or not to visualize the processed image. Default: True.
• save_path (str, optional) – The filepath(s) to save the processed image(s). If None, the image will not be saved. Default: None.

Returns:

interpretations/heatmap for images

Return type:

[numpy.ndarray]

Generic Attention

class interpretdl.GAInterpreter(model: callable, device: str = 'gpu:0')

Generic Attention Interpreter.

This is a specific interpreter for Bi-Modal Transformers models. GAInterpreter computes the attention map with gradients, and follows the operations that are similar to Rollout, with advanced modifications.

This implementation is suitable for models with self-attention in each modality, like CLIP.

More details regarding the Generic Attention method can be found in the original paper: https://arxiv.org/abs/2103.15679.

Parameters:

• model (callable) – A model with forward() and possibly backward() functions.
• device (str) – The device used for running model, options: "cpu", "gpu:0", "gpu:1" etc.

interpret(image_input: str, text: str, text_tokenized: numpy.ndarray, vis_attn_layer_pattern: str = '^visual.transformer.resblocks.[0-9]*.attn.attn_map$', txt_attn_layer_pattern: str = '^transformer.resblocks.[0-9]*.attn.attn_map$', start_layer: int = 11, start_layer_text: int = 11, resize_to: int = 224, crop_to: int = None, visual: bool = True, save_path: str = None) → tuple

Given image_input and text_tokenized, GAInterpreter first obtains all attention maps (of layers whose name matches vis_attn_layer_pattern or txt_attn_layer_pattern for visual and text modalities respectively) and their gradients of the prediction. Then, GAInterpreter computes the multiplication between the attention map and its gradient for each block, and obtains the matrix multiplication of all blocks. The start_layer controls the number of involved layers. The order of involving attention maps (from the first layer to the last) is the same as Rollout (from first to last).

Parameters:

• image_input (str or np.ndarray) – The input image filepath or a list of filepaths or numpy array of read images.
• text (str) – original texts, for visualization.
• text_tokenized (np.ndarray) – The tokenized text for the model’s input directly.
• vis_attn_layer_pattern (str, optional) – the pattern name of the layers whose features will output for visual modality. Defaults to '^visual.transformer.resblocks.*.attn.attn_map$'.
• txt_attn_layer_pattern (str, optional) – the pattern name of the layers whose features will output for text modality. Defaults to '^transformer.resblocks.*.attn.attn_map$'.
• start_layer (int, optional) – Compute the state from the start layer for visual modality. Defaults to 11.
• start_layer_text (int, optional) – Compute the state from the start layer for text modality. Defaults to 11.
• resize_to (int, optional) – Images will be rescaled with the shorter edge being resize_to. Defaults to 224.
• crop_to (int, optional) – After resize, images will be center cropped to a square image with the size crop_to. If None, no crop will be performed. Defaults to None.
• visual (bool, optional) – Whether or not to visualize the processed image. Default: True.
• save_path (str, optional) – The filepath(s) to save the processed image(s). If None, the image will not be saved. Default: None.

Returns:

(text_relevance: np.ndarray, image_relevance: np.ndarray)

Return type:

tuple

class interpretdl.GANLPInterpreter(model: callable, device: str = 'gpu:0', **kwargs)

Generic Attention Interpreter.

This is a specific interpreter for Bi-Modal Transformers models. GAInterpreter computes the attention map with gradients, and follows the operations that are similar to Rollout, with advanced modifications.

The following implementation is specially designed for Ernie.

More details regarding the Generic Attention method can be found in the original paper: https://arxiv.org/abs/2103.15679.

Parameters:

• model (callable) – A model with forward() and possibly backward() functions.
• device (str) – The device used for running model, options: "cpu", "gpu:0", "gpu:1" etc.

interpret(raw_text: str, tokenizer: callable = None, text_to_input_fn: callable = None, label: int = None, start_layer: int = 11, attn_map_name='^[a-z]*.encoder.layers.[0-9]*.self_attn.attn_drop$', gradient_of: str = 'logit', max_seq_len=128, visual=False)

Parameters:

• data (str or list of strs or numpy.ndarray) – The input text filepath or a list of filepaths or numpy array of read texts.
• start_layer (int, optional) – Compute the state from the start layer. Default: 11.
• label (list or tuple or numpy.ndarray, optional) – The target labels to analyze. The number of labels should be equal to the number of texts. If None, the most likely label for each text will be used. Default: None.
• attn_map_name (str, optional) – The layer name to obtain attention weights. Default: ^ernie.encoder.layers.*.self_attn.attn_drop$.
• gradient_of (str, optional) – compute the gradient of [‘probability’, ‘logit’ or ‘loss’]. Default: 'logit'. Multi-class classification uses probabitliy, while binary classification uses logit.

Returns:

interpretations for texts

Return type:

[numpy.ndarray]

class interpretdl.GACVInterpreter(model: callable, device: str = 'gpu:0', **kwargs)

The following implementation is specially designed for Vision Transformer.

Parameters:

• model (callable) – A model with forward() and possibly backward() functions.
• device (str) – The device used for running model, options: "cpu", "gpu:0", "gpu:1" etc.

interpret(inputs: str, start_layer: int = 3, attn_map_name='^blocks.[0-9]*.attn.attn_drop$', label: int = None, gradient_of: str = 'probability', resize_to: int = 224, crop_to: int = None, visual: bool = True, save_path: str = None)

Parameters:

• inputs (str or list of strs or numpy.ndarray) – The input image filepath or a list of filepaths or numpy array of read images.
• start_layer (int, optional) – Compute the state from the start layer. Default: 4.
• attn_map_name (str, optional) – The layer name to obtain attention weights. Default: ^blocks.*.attn.attn_drop$
• label (list or tuple or numpy.ndarray, optional) – The target labels to analyze. The number of labels should be equal to the number of images. If None, the most likely label for each image will be used. Default: None.
• gradient_of (str, optional) – compute the gradient of [‘probability’, ‘logit’ or ‘loss’]. Default: 'probability'. Multi-class classification uses probabitliy, while binary classification uses logit.
• resize_to (int, optional) – Images will be rescaled with the shorter edge being resize_to. Defaults to 224.
• crop_to (int, optional) – After resize, images will be center cropped to a square image with the size crop_to. If None, no crop will be performed. Defaults to None.
• visual (bool, optional) – Whether or not to visualize the processed image. Default: True.
• save_path (str, optional) – The filepath(s) to save the processed image(s). If None, the image will not be saved. Default: None.

Returns:

interpretations/heatmap for images

Return type:

[numpy.ndarray]

Bidirectional Transformer Interpreter

class interpretdl.BTCVInterpreter(model: callable, device: str = 'gpu:0', **kwargs)

Bidirectional Transformer Interpreter.

This is a specific interpreter for Transformers models, with two sub-processes: attentional perception, reasoning feedback.

The following implementation is specially designed for Vision Transformer.

Parameters:

• model (callable) – A model with forward() and possibly backward() functions.
• device (str) – The device used for running model, options: "cpu", "gpu:0", "gpu:1" etc.

interpret(inputs: str, ap_mode: str = 'head', start_layer: int = 3, steps: int = 20, attn_map_name='^blocks.[0-9]*.attn.attn_drop$', attn_v_name='^blocks.[0-9]*.attn.qkv$', attn_proj_name='^blocks.[0-9]*.attn.proj$', gradient_of: str = 'probability', label: int = None, resize_to: int = 224, crop_to: int = None, visual: bool = True, save_path: str = None)

Parameters:

• inputs (str or list of strs or numpy.ndarray) – The input image filepath or a list of filepaths or numpy array of read images.
• ap_mode (str, optional) – The approximation method of attentioanl perception stage, “head” for head-wise, “token” for token-wise. Default: head.
• start_layer (int, optional) – Compute the state from the start layer. Default: 4.
• steps (int, optional) – number of steps in the Riemann approximation of the integral. Default: 20.
• attn_map_name (str, optional) – The layer name to obtain the attention weights, head-wise/token-wise. Default: ^blocks.*.attn.attn_drop$.
• attn_v_name (str, optional) – The layer name for query, key, value, token-wise. Default: blocks.*.attn.qkv.
• attn_proj_name (str, optional) – The layer name for linear projection, token-wise. Default: blocks.*.attn.proj.
• gradient_of (str, optional) – compute the gradient of [‘probability’, ‘logit’ or ‘loss’]. Default: 'probability'. Multi-class classification uses probabitliy, while binary classification uses logit.
• label (list or tuple or numpy.ndarray, optional) – The target labels to analyze. The number of labels should be equal to the number of images. If None, the most likely label for each image will be used. Default: None.
• resize_to (int, optional) – Images will be rescaled with the shorter edge being resize_to. Defaults to 224.
• crop_to (int, optional) – After resize, images will be center cropped to a square image with the size crop_to. If None, no crop will be performed. Defaults to None.
• visual (bool, optional) – Whether or not to visualize the processed image. Default: True.
• save_path (str, optional) – The filepath(s) to save the processed image(s). If None, the image will not be saved. Default: None.

Returns:

interpretations/heatmap for images

Return type:

[numpy.ndarray]

class interpretdl.BTNLPInterpreter(model: callable, device: str = 'gpu:0', **kwargs)

Bidirectional Transformer Interpreter.

This is a specific interpreter for Transformers models, with two sub-processes: attentional perception, reasoning feedback.

The following implementation is specially designed for Ernie.

Parameters:

• model (callable) – A model with forward() and possibly backward() functions.
• device (str) – The device used for running model, options: "cpu", "gpu:0", "gpu:1" etc.

interpret(raw_text: str, tokenizer: callable = None, text_to_input_fn: callable = None, label: list = None, ap_mode: str = 'head', start_layer: int = 11, steps: int = 20, embedding_name='^[a-z]*.embeddings$', attn_map_name='^[a-z]*.encoder.layers.[0-9]*.self_attn.attn_drop$', attn_v_name='^[a-z]*.encoder.layers.[0-9]*.self_attn.v_proj$', attn_proj_name='^[a-z]*.encoder.layers.[0-9]*.self_attn.out_proj$', gradient_of: str = 'logit', max_seq_len=128, visual=False)

Parameters:

• data (str or list of strs or numpy.ndarray) – The input text filepath or a list of filepaths or numpy array of read texts.
• ap_mode (str, default to head-wise) – The approximation method of attentioanl perception stage, “head” for head-wise, “token” for token-wise. Default: head.
• start_layer (int, optional) – Compute the state from the start layer. Default: 11.
• steps (int, optional) – number of steps in the Riemann approximation of the integral. Default: 20.
• embedding_name (str, optional) – The layer name for embedding, head-wise/token-wise. Default: ^ernie.embeddings$.
• attn_map_name (str, optional) – The layer name to obtain the attention weights, head-wise/token-wise. Default: ^ernie.encoder.layers.*.self_attn.attn_drop$.
• attn_v_name (str, optional) – The layer name for value projection, token-wise. Default: ^ernie.encoder.layers.*.self_attn.v_proj$.
• attn_proj_name (str, optional) – The layer name for linear projection, token-wise. Default: ernie.encoder.layers.*.self_attn.out_proj$.
• gradient_of (str, optional) – compute the gradient of [‘probability’, ‘logit’ or ‘loss’]. Default: 'logit'. Multi-class classification uses probabitliy, while binary classification uses logit.
• label (list or tuple or numpy.ndarray, optional) – The target labels to analyze. The number of labels should be equal to the number of texts. If None, the most likely label for each text will be used. Default: None.

Returns:

interpretations for texts

Return type:

[numpy.ndarray]