Evaluating Top-N Recommendations Using Ranked Error Approach: An Empirical Analysis

Evaluation metrics or measures are necessary tools for evaluating and choosing the most appropriate modeling approaches within recommender systems. However, evaluation measures can sometimes fall short when evaluating recommendation lists that best match users' top preferences. A possible reason for this shortcoming is that most measures mainly focus on the list-wise performance of the recommendations and generally do not consider the item-wise performance. As a result, a recommender system might apply a weak or less accurate modeling approach instead of the best one. To address these challenges, we propose a new evaluation measure that incorporates the rank order of a prediction list with an error-based metric to make it more powerful and discriminative and thus more suited for top-N recommendations. The main goal of the proposed metric is to provide recommender systems, developers and researchers an even better tool, which enables them to choose the best modeling approach possible, and hence maximizing the quality of top-N recommendations. To evaluate the proposed metric and compare its general properties against existing metrics, we perform extensive experiments with detailed empirical analysis. Our experiments and the analysis show the usefulness, effectiveness and feasibility of the new metric.

Automated summary

Evaluation metrics are important in recommender systems for choosing appropriate modeling approaches, but they may fall short when evaluating recommendation lists that match users' top preferences. A new evaluation measure is proposed to address this challenge by incorporating rank order of prediction lists with an error-based metric, providing a powerful tool for selecting the best modeling approach for maximizing the quality of top-N recommendations. Extensive experiments show the usefulness, effectiveness, and feasibility of the new metric compared to existing ones.