EEMCS EPrints Service
Demeester, T. and Aly, R.B.N. and Hiemstra, D. and Nguyen, Dong-Phuong and Trieschnigg, R.B. and Develder, C. (2014) Exploiting User Disagreement for Web Search Evaluation: an Experimental Approach. In: Proceedings of the 7th ACM International Conference on Web Search and Data Mining, WSDM 2014, 24-28 Feb 2014, New York, USA. pp. 33-42. ACM. ISBN 978-1-4503-2351-2
Full text available as:
Official URL: http://dx.doi.org/10.1145/2556195.2556268
To express a more nuanced notion of relevance as compared to binary judgments, graded relevance levels can be used for the evaluation of search results. Especially in Web search, users strongly prefer top results over less relevant results, and yet they often disagree on which are the top results for a given information need. Whereas previous works have generally considered disagreement as a negative effect, this paper proposes a method to exploit this user disagreement by integrating it into the evaluation procedure. First, we present experiments that investigate the user disagreement. We argue that, with a high disagreement, lower relevance levels might need to be promoted more than in the case where there is global consensus on the top results. This is formalized by introducing the User Disagreement Model, resulting in a weighting of the relevance levels with a probabilistic interpretation. A validity analysis is given, and we explain how to integrate the model with well-established evaluation metrics. Finally, we discuss a specific application of the model, in the estimation of suitable weights for the combined relevance of Web search snippets and pages.
Export this item as:
To correct this item please ask your editor
Repository Staff Only: edit this item