Data Scrambling: All the Synthetic Data You'll Ever Need

Data from scrambled continuous contracts are termed synthetic data be­cause these data are not from actual trading in the open market. The phrase data scrambling is used because this method randomly rearranges the data to create new sequences.

Let us first see how to encapsulate market information. We place two daily bars next to each other. Then, we observe the relationship be­tween the open (0), high (H), low (L), and close (C) of the second bar by using the close of the first bar as the reference. We can write the re­lationship as follows:

deltaO = 0 - C[1], dekaH = H - C[1], deltaL = L - C[1], dehaC = C - C[1].

Here [1] denotes the close of the previous day. These equations en­capsulate market trading behavior since they capture price patterns ver­sus the previous close. Over a period of years, each market will have some characteristic values for these equations, based on its volatility, li­quidity, and other trading patterns. When we sample with replacement using these formulas, we create patterns that bear the market's signature as defined by relative price relationships.

The next step is to use a random number generator to scramble the bars. Once you have a new sequence, you need a starting point, usually the prior close. You. can use any number for the first bar. The new bar is derived from the prior close as follows (new synthetic values are indi­cated by the Syn prefix):

Syn-Close = Close [1] + delta C, Syn-High = Close[l] + delta H,

232 Data Scrambling

Syn-Low = Close [1] + delta L, Syn-Open = Close[l] + delta 0.

So the calculations are easy to put into a program or a spreadsheet. We will first calculate the interbar relationships in Table 8.2, which is based on the actual bar sequence of the December 1995 S&P-500 (Fig­ure 8.1). The interbar relationships appear in the last four columns. The difference between the daily close of 11/30 and 11/29 was -0.80 points. The differences between yesterday's close and today's open, high, low, and close are shown for each day. These calculations encapsulate price relationships. Now we can scramble these bars using a random number generator.

To scramble these data, number them from 1 to n, where n is the last bar. Then, use the random number generator to pick a number be­tween 1 and n. That number is the next bar in the sequence. Suppose on the tenth pick, you pick bar 5. Then the original bar 5 becomes bar 10 of the new sequence. The bars may repeat more than once. For example, on the twenty-seventh pick, you may draw bar 5 once again. You can generate as long a sequence as you desire.

Here we used the 11/29/95 close of 608.05 as reference, and gen­erated a new sequence of bars using the sampling function in Microsoft Excel. The new sequence was: 4, 5, 8, 1, 3, 10, 10, 8, 9, 1. Therefore, starting with the previous close of 608.05, we put in the fourth bar of the original data, then the fifth bar, and the eighth bar, and so on.

Table 8.3, page 234, presents the spreadsheet used to calculate the new synthetic data. The first column is the bar number drawn by sam-

Table 8.2 Spreadsheet based on December 1995 SScP-500 data.

0- H- L- C-Bar# Date Open High Low Close C[1] C[1] C[1] C[1]

11/29/95 608.6 608.85 606.3 608.05

1 11/30/95 608.07 610 606.1 607.25 0.02 1.95 -1.95 -0.8

2 12/01/95 608 609.4 605.9 608.3 0.75 2.15 -1.35 1.05

3 12/04/95 608.87 615.5 608.05 614.8 0.57 7.2 -0.25 6.5

4 12/05/95 614.02 619.5 613.85 618.75 -0.78 4.7 -0.95 3.95

5 12/06/95 619.85 622.65 617.35 619.8 1.1 3.9 -1.4 1.05

6 12/07/95 618.95 619.8 615.7 616.65 -0.85 0 -4.1 -3.15

7 12/08/95 618.5 619.5 614.3 618.3 1.85 2.85 -2.35 1.65

8 12/11/95 618.9 621.7 617.75 619.8 0.6 3.4 -0.55 1.5

9 12/12/95 618.9 620 618.2 618.8 -0.9 0.2 -1.6 -1

10 12/13/95 619.4 622.6 618.65 621.35 0.6 3.8 -0.15 2.55

Data Scrambling: All the Synthetic Data You'll Ever Need 233

SP Dec 95 contract dally bars

1 2 3 4 5 6 7 8 9 10 11/30/95-12/13/95

Figure 8.1 Actual price bars from the December 1995 SScP-500 contract.

pling with replacement. The next four columns are the inter-bar rela­tions previous calculated for each bar in Table 8.2. The last four columns are the synthetic data derived from the previous close by adding the in-terbar relations.

In Table 8.2, the data for 12/05 converts to bar 4, and the market gained 3.95 points on the close. In Table 8.3, bar 4 is the first bar of the new sequence. The previous close was assumed to be 608.05. So the new close is 3.95 + 608.05, or 612.00. The new low is 608.05 - 0.95, or 607.10. The new high is 4.7 + 608.05, or 612.75. These are the numbers for the synthetic contract in the first row. The close for the second bar is 612.00 + 1.05 or 613.05. You can now complete the rest of the calcu­lations. The new scrambled data created a synthetic bar pattern plotted in Figure 8.2.

The new bar sequence (Figure 8.2) shows an upward bias, getting up toward the 630 area. Note how it does not show the consolidation in the last six bars of the original data. The last bar from the original data (bar 10 on Figure 8.1) occurs in Figure 8.2 as bars 6 and 7, and you can see that the relative appearance of the high and low versus the previous

234 Data Scrambling

Table 8.3 Scrambled data for SSiP-500 using relationships calculated in Table 8.2. I assumed that the close before the first bar (bar 4 below) was 608.05.

Bar #	0- C[1]	H- C[1]	L- C[1]	C- C[1]	Syn-0	Syn-H	Syn-L	Syn-C
	-0.78	4.7	-0.95	3.95	607.27	612.75	607.1
	1.1	3.9	-1.4	1.05	613.1	615.9	610.06	613.05
	0.6	3.4	-0.55	1.5	613.65	616.45	612.5	614.55
	0.02	1.95	-1.95	-0.8	614.57	616.5	612.6	613.75
	0.57	7.2	-0.25	6.5	614.32	620.95	613.5	620.25
	0.6	3.8	-0.15	2.55	620.85	624.05	620.1	622.8
	0.6	3.8	-0.15	2.55	623.4	626.6	622.65	625.35
	0.6	3.4	-0.55	1.5	625.95	628.75	624.8	626.85
	-0.9	0.2	-1.6	-1	625.95	627.05	625.25	625.85
	0.02	1.95	-1.95	-0.8	625.87	627.8	623.9	625.05

Scrambled Data SP futures contract

1 2 3 4 5 6 7 8 9 10

Time(days)

Figure 8.2 Synthetic data from Table 8.3 for the S&P-500 contract.

Data Scrambling: All the Synthetic Data You'll Ever Need 235

bar in the sequence is similar in both cases. Thus, we have encapsulated the market behavior in the original bar 10 and reproduced it in another sequence to create new synthetic data.

Of course, the more samples you generate, the greater the variety of patterns you will see. Another synthetic pattern derived from original December 1995 S&P-500 futures data is shown in Figure 8.3 to illus­trate the variety of possible patterns. The new patterns cover a narrower price range than the original data, and appear like a breakout at the end.

Thus, you can generate a variety of chart patterns using data scrambling. Continuous contracts work well to create synthetic data be­cause they represent a long history of market action. If you prefer, you can use this method on individual contracts, and then string them to­gether for testing using rollovers at the appropriate dates.

The power of data scrambling increases as the number of bars of data increases because you can create a greater variety of patterns. Thus, if you had, say, a 5-year or 7-year long continuous contract, you could