Contrasting Neural Click Models and Pointwise IPS Rankers

Inverse-propensity scoring and neural click models are two popular methods for learning rankers from user clicks that are affected by position bias. Despite their prevalence, the two methodologies are rarely directly compared on equal footing. In this work, we focus on the pointwise learning setting to compare the theoretical differences of both approaches and present a thorough empirical comparison on the prevalent semi-synthetic evaluation setup in unbiased learning-to-rank. We show theoretically that neural click models, similarly to IPS rankers, optimize for the true document relevance when the position bias is known. However, our work also finds small but significant empirical differences between both approaches indicating that neural click models might be affected by position bias when learning from shared, sometimes conflicting, features instead of treating each document separately.

Automated summary

Inverse-propensity scoring and neural click models are compared in this study for learning rankers from user clicks affected by position bias. Theoretical differences are explored and empirical comparisons are conducted on a prevalent evaluation setup. It is shown that both methods optimize for true document relevance when position bias is known, but small empirical differences are found when neural click models learn from shared, conflicting features.