Laboratory Experiments in Information Retrieval by Tetsuya Sakai

Author:Tetsuya Sakai
Language: eng
Format: epub
ISBN: 9789811311994
Publisher: Springer Singapore

B

p-value

5,000

0.2018

10,000

0.2016

20,000

0.2030

50,000

0.2060

100,000

0.2059

200,000

0.2043

500,000

0.2041

1,000,000

0.2040

4.5.2 Randomised Tukey HSD Test

Figure 4.12 explains the concept of the RTHSD test for m = 3 systems, where we are interested in the difference between every system pair. We start with an observed matrix U shown in the top left corner; note that the data structure is the same as Fig. 4.3 from Sect. 4.4.3. As with the case with m = 2 systems, our null hypothesis is that the observed scores actually come from the same systems; thus, if we observe (x 1j, x 2j, x 3j), then other assignments such as (x 3j, x 2j, x 1j) and (x 2j, x 3j, x 1j) must have been equally likely to occur. Hence we create B replicates of the original matrix, U ∗b (b = 1, …, B). For each replicate U ∗b, we compute the new sample means as before, and compute the maximum between-system difference , to build a null hypothesis distribution shown in the bottom right of Fig. 4.12. Note that the above null distribution represents the difference between the best possible and the worst possible systems. Finally, we compare each observed between-system difference with the null distribution. Note that this procedure follows the logic of the original Tukey HSD test to ensure that the familywise error rate is α; the only difference is how we obtain the null distribution.

Fig. 4.12Principle of the RTHSD test for a given topic-by-run matrix

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.