How Suitable Is Your Naturalistic Dataset for Theory-based User Modeling?

Theory-based, or white-box, models come with a major benefit that makes them appealing for deployment in user modeling: their parameters are interpretable. However, most theory-based models have been developed in controlled settings, in which researchers determine the experimental design. In contrast, real-world application of these models demands setups that are beyond developer control. In non-experimental, naturalistic settings, the tasks with which users are presented may be very limited, and it is not clear that model parameters can be reliably inferred. This paper describes a technique for assessing whether a naturalistic dataset is suitable for use with a theory-based model. The proposed parameter recovery technique can warn against possible over-confidence in inferred model parameters. This technique also can be used to study conditions under which parameter inference is feasible. The method is demonstrated for two models of decision-making under risk with naturalistic data from a turn-based game.

Automated summary

Theory-based models have interpretable parameters, but their inference can be challenging in real-world applications. This paper proposes a technique to assess the applicability of naturalistic datasets and demonstrates its use in two decision-making models under risk.