A Storey q-value measures the smallest positive false discovery rate associated with a nested rejection region that includes a given test result. Its practical estimate turns a family of p-values into a ranked discovery shortlist while estimating the fraction of true null hypotheses. For a quant screening many prespecified candidate signals, that adaptation can select more candidates than ordinary Benjamini–Hochberg, with additional estimation assumptions to justify. The population quantity and its estimate should be distinguished.
Which false discovery rate?
Let be the number of false rejections and the total number of rejections. The false discovery rate is . The positive false discovery rate conditions on making at least one discovery:
For p-value threshold regions, Storey's population definition is
where the infimum is over thresholds in for which the conditional expectation is defined. Every such region includes the observed result. This is a property of rejection regions under a probability model, not an observed count of mistakes. Storey (2003), Definition 2, gives the nested-region definition.
Estimate the null fraction, then work backwards
Write for the unknown proportion of true nulls among tests. Under a continuous uniform null-p-value model, a true null lands above a fixed tuning threshold with probability . Counting that upper tail therefore gives a way to estimate . Alternative hypotheses can also produce large p-values: tail membership does not identify which hypotheses are null. A high leaves fewer observations and can make the estimate unstable. The official qvalue documentation discusses the null-p-value assumption and estimation choices.
For the hand calculation below, use this fixed-, finite-sample plus-one convention:
The added one and the cap at one are explicit choices. Sort the p-values in increasing order, retaining their hypothesis identifiers, and calculate the practical estimates
Start at the largest rank and take a running minimum backwards. This makes the estimated q-values nondecreasing with the sorted p-values; a more significant test cannot receive a larger value than a less significant one. Setting the multiplier to one gives the usual BH adjusted p-values. The authors' qvalue source implements the backward cumulative-minimum principle.
Ten hypothetical signal tests
Suppose the ten p-values below come from a prespecified family of correctly calibrated tests. They are stipulated numbers, with no known truth labels. Fix before inspecting them. Only and exceed , so
| Sorted rank | Observed p-value | Estimated q-value | BH adjusted p-value |
|---|---|---|---|
| 1 | 0.001 | 0.006 | 0.010 |
| 2 | 0.005 | 0.015 | 0.025 |
| 3 | 0.010 | 0.020 | 0.033333 |
| 4 | 0.020 | 0.030 | 0.050 |
| 5 | 0.030 | 0.036 | 0.060 |
| 6 | 0.040 | 0.040 | 0.066667 |
| 7 | 0.080 | 0.068571 | 0.114286 |
| 8 | 0.200 | 0.150 | 0.250 |
| 9 | 0.600 | 0.400 | 0.666667 |
| 10 | 0.800 | 0.480 | 0.800 |
Values are rounded for display; decisions use unrounded values. In this example the raw are already increasing, so the backward minimum leaves them unchanged.
At a selection level of , the rule selects the first six tests. The sixth has ; the seventh has . Ordinary BH at selects four, including the boundary adjusted value . The adaptive shortlist ends at p-value , below the fixed .
Original fixed-lambda arithmetic illustration. Solid circles show estimated q-values and dashed squares show BH adjusted p-values. The two additional selections have unknown truth labels.
What the shortlist does not certify
An estimated q-value of does not assign that individual hypothesis a 4% posterior probability of being null. Local FDR estimates address the null probability conditional on a particular result under a fitted mixture model; q-values concern nested rejection regions. Storey's Bayesian interpretation likewise conditions on membership in a region, which is a different event. The qvalue vignette, §§5.3.4–5.3.5, separates q-values from local FDR.
Nor does selecting at cap the realised false-discovery proportion at 5%. FDR control, when its assumptions hold, concerns an expectation across repeated studies. The arithmetic above establishes the selections, not their truth or an FDR theorem for this plug-in rule at every level.
This small example is not a reproduction of a default software run. The package uses smoothing by default to estimate , offers a bootstrap method and an optional pFDR adjustment, and its single- estimator does not use the plus-one numerator shown here. Record the estimator, tuning choices and software options when reporting results.
Adaptive FDR guarantees require the conditions of the particular procedure. Arbitrary plug-in estimates, data-driven tuning and arbitrary dependence are not automatically covered. Shared market histories can create dependence between tests; invalid individual p-values or an undisclosed strategy search are separate problems. For an arbitrary-dependence FDR procedure with valid null p-values, see Benjamini–Yekutieli.
For the outcome table, FDR definition and ordinary BH baseline, use Multiple Testing and the False Discovery Rate (public preview; full lesson requires membership).
Practice: why take a running minimum?
Four hypothetical sorted p-values are . Fix and use the same plus-one convention. A colleague computes raw values and drops the second test at selection level . Is that the q-value rule?
Answer. No. One p-value exceeds , so . Taking the backwards running minimum gives . All first three tests are selected at . The third-rank region includes the second test too, so assigning the second a larger q-value would ignore that qualifying region.