Toward Practical Usage of the Attention Mechanism as a Tool for Interpretability

Natural language processing (NLP) has been one of the subfields of artificial intelligence much affected by the recent neural revolution. Architectures such as recurrent neural networks (RNNs) and attention-based transformers helped propel the state of the art across various NLP tasks, such as sequence classification, machine translation, and natural language inference. However, if neural models are to be used in high-stakes decision making scenarios, the explainability of their decisions becomes a paramount issue. The attention mechanism has offered some transparency in the workings of otherwise black-box RNN models: attention weights (scalar values assigned input words) invite to be interpreted as the importance of that word, providing a simple method of interpretability. Recent work, however, has questioned the faithfulness of this practice. Subsequent experiments have shown that faithfulness of attention weights may still be achieved by incorporating word-level objectives in the training process of neural networks. In this article, we present a study that extends the techniques for improving faithfulness of attention based on regularization methods that promote retention of word-level information. We perform extensive experiments on a wide array of recurrent neural architectures and analyze to what extent the explanations provided by inspecting attention weights are correlated with the human notion of importance. We find that incorporating tying regularization consistently improves both the faithfulness (-0.14 F1, +0.07 Brier, on average) and plausibility (+53.6% attention mass on salient tokens) of explanations obtained through inspecting attention weights across analyzed datasets and models.

Automated summary

Natural language processing (NLP) has been greatly influenced by the neural revolution in artificial intelligence. Attention mechanisms provide transparency to previously black-box recurrent neural network (RNN) models, but recent research questions their faithfulness. This study presents a regularization technique to improve the faithfulness of attention-based explanations, showing consistent improvements across various datasets and models.