Multiple comparisons problem

An example of coincidence produced by data dredging (uncorrected multiple comparisons) showing a correlation between the number of letters in a spelling bee's winning word and the number of people in the United States killed by venomous spiders. Given a large enough pool of variables for the same time period, it is possible to find a pair of graphs that show a spurious correlation.

Multiple comparisons, multiplicity or multiple testing problem occurs when many statistical tests are performed on the same dataset. Each test has its own chance of a Type I error (false positive), so the overall probability of making at least one false positive increases as the number of tests grows. In statistics, this occurs when one simultaneously considers a set of statistical inferences or estimates a subset of selected parameters based on observed values.

The probability of false positives is measured through the family-wise error rate (FWER). The larger the number of inferences made in a series of tests, the more likely erroneous inferences become. Several statistical techniques have been developed to compensate for the number of inferences being made—for example, by requiring a stricter significance threshold for individual comparisons.