Temporal logic explanations for dynamic decision systems using anchors and Monte Carlo Tree Search

For many automated perception and decision tasks, state-of-the-art performance may be obtained by algorithms that are too complex for their behavior to be completely understandable or predictable by human users, e.g., because they employ large machine learning models. To integrate these algorithms into safety-critical decision and control systems, it is particularly important to develop methods that can promote trust into their decisions and help explore their failure modes. In this article, we combine the anchors methodology with Monte Carlo Tree Search to provide local model-agnostic explanations for the behaviors of a given black-box model making decisions by processing time-varying input signals. Our approach searches for descriptive explanations for these decisions in the form of properties of the input signals, expressed in Signal Temporal Logic, which are highly likely to reproduce the observed behavior. To illustrate the methodology, we apply it in simulations to the analysis of a hybrid (continuous-discrete) control system and a collision avoidance system for unmanned aircraft (ACAS Xu) implemented by a neural network.(c) 2023 Elsevier B.V. All rights reserved.

Automated summary

For complex automated perception and decision tasks, it is important to develop trust and understand the behavior of algorithms that may be too complex for human users. This article combines the anchors methodology with Monte Carlo Tree Search to provide explanations for black-box models' decisions. The methodology searches for descriptive explanations in the form of input signal properties, expressed in Signal Temporal Logic, to reproduce observed behavior.