Statistical Significance of Channel Breakout Variations

One of the key questions you must ask when you see system variations is whether the differences in the performance are statistically significant. There are many ways to differentiate systems other than comparing

162 Developing Trading System Variations

their net profits. For example, one system may produce fewer trades. Another system may produce a smaller drawdown. Some systems may have a higher percentage of profitable trades. A variation may be useful precisely because it suits your trading style.

A paired t-test compares the means of two system variables meas­ured on the same data sets. This test will tell you if the means of the two series being compared are truly different in a statistically significant sense. You can find a detailed discussion of the t-test in any book on sta­tistics. For our purpose, the test gives two numbers: a t-statistic for the data and an appropriate reference value (t-critical). If our t-statistic is greater than t-critical, then the difference in performance is statistically significant. The Microsoft Excel 5.0 spreadsheet has a built-in t-test in the Tools menu under Data analysis.

We need a reference or standard system for comparison before we can use the t-test. Here we use a 20-day channel breakout system with­out any exits as the reference system. If you do not use any specific exits, you have a system in which long entries are short exits and vice versa. Thus, the effects of adding exit strategies should be easy to measure against this reference system. Table 5.8 summarizes the results of testing the 20-day channel breakout system without any exits on the previous data. As usual, there is a $1,500 initial stop and $100 allowance for slip­page and commissions.

Let us examine the effects of adding the 5-day high-low trailing stop to the reference system (see Table 5.2). In comparing the net profits from each system, the t-statistic was 2.224 and the t-critical value was 2.262 with 9 degrees of freedom. Thus, the t-statistic was smaller than the reference t-critical value. The average profit for the reference system is $51,368 versus $27,949 with the trailing exit. At the 5-percent level, this difference barely missed being significant. Hence, adding the exit probably significantly reduced average profits. The spreadsheet calcula­tions are given in Table 5.9.

Table 5.10, page 164, compares the number of trades from each system. The average number of trades was 220 with the trailing exit (see Table 5.2), and 118 with the reference system (see Table 5.8). The t-statistic for the number of trades was 8.05. The t-critical value is 2.262, confirming that there is a statistically significant increase in the number of trades. Therefore, the total number of trades increases sig­nificantly when we add a trailing exit.

You can now combine the results from these two comparisons. Those results say that adding a trailing exit at the 5-day high-low signifi­cantly reduced profits by 30 percent while increasing the number of

Statistical Significance of Channel Breakout Variations 163 Table 5.8 Historical results for 20-bar breakout on the close without any exits

Table 5.9 A summary of the t-test comparing the reference system to the CHBOC with 5-day trailing stop