Consensus-based evaluation framework for distributed information retrieval systems

Abstract

Multi-agent systems have been attacking the challenges of information retrieval tasks on distributed environment. In this paper, we propose a consensus choice selection method based framework to evaluate the performance of cooperative information retrieval tasks of the multiple agents. Thereby, two well-known measurements, precision and recall, are extended to handle consensual closeness (i.e., local and global consensus) between the sets of retrieved results. We show that in a motivating example the proposed criteria are prone to solve the rigidity problem of classical precision and recall. More importantly, the retrieved results can be ranked with respect to the consensual score, and the ranking mechanism has been verified to be more reasonable.