  We present a toolkit to facilitate the interpretation and understanding of
neural network models. The toolkit provides several methods to identify salient
neurons with respect to the model itself or an external task. A user can
visualize selected neurons, ablate them to measure their effect on the model
accuracy, and manipulate them to control the behavior of the model at the test
time. Such an analysis has a potential to serve as a springboard in various
research directions, such as understanding the model, better architectural
choices, model distillation and controlling data biases.

Computation and Language