Practical significance asks whether an effect matters for a decision. A return example separates evidence of a gross edge from its value after costs and an economic hurdle.
Practical significance
Practical significance is the importance of an effect for a real decision, judged against the costs, risks, constraints and objectives of that decision. In quantitative finance, a return advantage can be measured precisely and still be too small to justify trading. Conversely, a potentially valuable advantage may remain too uncertain to act on.
Statistical significance concerns evidence against a specified hypothesis under a sampling model. Practical significance concerns what the effect would change. A p-value supplies neither the size of the benefit nor a trading rule. The American Statistical Association's principles explicitly separate statistical significance from effect size and importance.
Start with the hurdle
For a trading strategy, put the gross return advantage and the costs on the same horizon and notional basis. A simple accounting model is
μnet=μgross−c,
where c is the all-in cost per completed round trip and each mean is a return per round trip. Break-even requires the true gross mean to exceed c. An investor may require an additional net hurdle h to compensate for risk, operational work or the opportunity cost of capital; the resulting economic target is μgross>c+h. That hurdle is a decision input, not something a significance test discovers.
The relevant effect need not be a return. A forecast improvement matters if it changes allocations or execution enough to improve the objective. A risk model improvement might matter because it reduces exposure to a damaging loss. There is no universal numerical cutoff for practical significance.
A clear gross result, an unattractive trade
Consider a hypothetical strategy with 10,000 independent, normally distributed round-trip returns, a known population standard deviation of 100 basis points per round trip, and an observed gross sample mean of 5 basis points. The strategy and analysis were fixed before observing the data. One basis point is 0.01 percentage points; returns and costs below use the same notional.
Its standard error is 100/10,000=1 basis point. Against a zero gross mean, the normal test statistic is z=5/1=5, giving a two-sided p-value of approximately 5.73×10−7. That is strong evidence against zero gross mean within this stipulated model.
Now stipulate a fixed all-in cost of 8 basis points per round trip, covering the cost categories applicable to this strategy without double-counting them. The estimated net mean is
μnet=5−8=−3 basis points per round trip.
Hypothetical signed return ledger: gross mean plus 5 basis points per round trip, cost deduction minus 8 basis points, net mean minus 3 basis points.
Original hypothetical calculation on a common round-trip and notional basis. The cost is stipulated and fixed, rather than estimated from market data.
A 95% normal confidence interval for the gross mean is approximately 3.04 to 6.96 basis points. Subtracting the fixed cost shifts it to −4.96 to −1.04 basis points for the net mean. Under these assumptions, even the upper endpoint is below break-even. More precise evidence of a positive gross effect would not make the fixed cost disappear.
The confidence level describes repeated-sampling coverage of the interval procedure, not a probability that the realised interval contains the mean. The NIST explanation of mean intervals develops this distinction. For the test and interval calculations in a broader course setting, Confidence Intervals and Hypothesis Testing has a public preview; the full lesson requires membership.
Make the uncertainty decision-relevant
Report the estimated effect in economic units, its uncertainty, and the hurdle it must clear. For a predeclared conservative decision screen with fixed costs, one might require a suitable lower confidence bound for the gross mean to exceed c+h. Specify the bound's confidence level and one-sided or two-sided convention before seeing results. Passing that screen still leaves risk, capacity and implementation decisions.
Costs are rarely known constants in a real backtest. Spread, fees, slippage, market impact and financing can depend on turnover, order size and market conditions. If costs are estimated or correlated with returns, shifting a gross-return interval by a single average cost generally fails to capture all net-return uncertainty. Likewise, overlapping or dependent trades can invalidate the independent-observation standard error used above.
A large estimate with a wide interval is potentially important but uncertain. A small estimate with a tight interval can be precisely measured but commercially irrelevant. Failure to reject a zero effect does not establish that every economically important effect has been ruled out; inspect the interval relative to the hurdle. Practical significance makes the decision explicit while keeping uncertainty visible.
