Synthetic Data

Generating synthetic market data that retains the same underlying characteristics is the easiest way to quickly validate if your algorithm has a real edge or if it's just overfit to historical data. Our market modifier engine allows you to create an unlimited amount of synthetic data. You can apply these modifiers during walk-forward testing to check the robustness of your algorithm.

Modifiers

Gap Fix

Most markets aren't open 24/7 and experience gaps in the data, whether in time or price. Think overnight closures, weekend gaps, or even futures contract rollovers. Unless your algorithm is specifically designed to trade these gaps, they can create misleading signals and distort your analysis.

Why are gaps a problem?

1. Indicators Misbehave: Smoothing indicators, oscillators, and volatility indicators can all produce inaccurate readings due to gaps.
2. False Signals: Gaps can create the illusion of trends or reversals where none exist.

How Spectral Alpha helps:

By default, Spectral Alpha uses a "back propagation" technique to fix gaps larger than 1% in the open-to-close price. This ensures that the latest prices are correct, while historical prices after a gap are adjusted accordingly. The relative price movements are preserved, only the absolute levels change.

Oversampling

Oversampling shifts the closing time of bars, allowing you to generate different charts from the same underlying data. This helps you explore how your algorithm performs under slightly different market conditions.

How it works:

• Time-based bars: Shifts the closing time backwards (e.g., a 4-hour bar closing at 00:00 might be shifted to 22:00).
• Volatility-based bars: Shifts the volatility trigger up or down.

Why it's useful:

Oversampling is used in our optimization engine to generate more trades from the same dataset. This helps you find robust parameters that are less likely to overfit.

Jitter

Jitter adds a small amount of random noise to the market data by shuffling bars locally with a low probability. This simulates the unpredictable nature of markets and helps you assess your algorithm's resilience to unexpected price fluctuations.

Invert

The invert modifier flips the market data, creating a mirror image of the price action. This can be useful for testing how your algorithm performs in different market conditions.

Two inversion methods:

1. Flat Inversion: Inverts the data around the mean price, which can change the direction of the underlying trend.
2. Linear Regression Line: Inverts the data around the line of best fit, preserving the overall trend direction.

Shuffle

The shuffle modifier disrupts any patterns or inefficiencies in the data by rearranging the bars. This helps you determine if your algorithm is truly capturing an edge or just exploiting random noise.

Two shuffling methods:

1. Without Replacement: Divides the data into chunks and rearranges them, maintaining the overall trend and volatility.
2. With Replacement: Randomly selects bars from the entire dataset, potentially using the same bar multiple times. This can create entirely new data with different characteristics.

Detrend

Detrend removes the overall trend from the data by adjusting the prices to flatten the slope of the linear regression line. This is useful for isolating and analyzing other market dynamics, such as volatility or cyclicality.

Combinations

You can combine multiple modifiers to create even more dramatic transformations of the market data. This allows you to explore a wide range of scenarios and rigorously test your algorithm's robustness.

Example:

Combining shuffling (without replacement) and detrending creates a dataset with no inter-bar inefficiencies. This can be used to validate if your algorithm is truly capturing an edge or simply fitting to noise in the data.