Home > Publications
Home University of Twente
Prospective Students
Intranet (internal)

EEMCS EPrints Service

25653 Exploiting User Disagreement for Web Search Evaluation: an Experimental Approach
Home Policy Brochure Browse Search User Area Contact Help

Demeester, T. and Aly, R.B.N. and Hiemstra, D. and Nguyen, Dong-Phuong and Trieschnigg, R.B. and Develder, C. (2014) Exploiting User Disagreement for Web Search Evaluation: an Experimental Approach. In: Proceedings of the 7th ACM International Conference on Web Search and Data Mining, WSDM 2014, 24-28 Feb 2014, New York, USA. pp. 33-42. ACM. ISBN 978-1-4503-2351-2

Full text available as:


215 Kb
Open Access

Official URL:

Exported to Metis


To express a more nuanced notion of relevance as compared to binary judgments, graded relevance levels can be used for the evaluation of search results. Especially in Web search, users strongly prefer top results over less relevant results, and yet they often disagree on which are the top results for a given information need. Whereas previous works have generally considered disagreement as a negative effect, this paper proposes a method to exploit this user disagreement by integrating it into the evaluation procedure. First, we present experiments that investigate the user disagreement. We argue that, with a high disagreement, lower relevance levels might need to be promoted more than in the case where there is global consensus on the top results. This is formalized by introducing the User Disagreement Model, resulting in a weighting of the relevance levels with a probabilistic interpretation. A validity analysis is given, and we explain how to integrate the model with well-established evaluation metrics. Finally, we discuss a specific application of the model, in the estimation of suitable weights for the combined relevance of Web search snippets and pages.

Item Type:Conference or Workshop Paper (Full Paper, Talk)
Research Group:EWI-HMI: Human Media Interaction, EWI-DB: Databases
Research Program:CTIT-NICE: Natural Interaction in Computer-mediated Environments
Research Project:AXES: Acces To Audiovisual Archives, COMMIT/Infiniti: Information Retrieval for Information Services
ID Code:25653
Deposited On:04 February 2015
More Information:statisticsmetis

Export this item as:

To correct this item please ask your editor

Repository Staff Only: edit this item