WFA Bug marked as fixed in 116 was NOT fixed
This is the bug: https://roadmap.strategyquant.com/tasks/sq4_3709
Now, the WFA table shows the correct dates for IS and OOS, but the last day of IS is still used as the first OOS day, EVEN THOUGH THE TABLE SAYS IT'S NOT.
You can verify this by running the strategy in Retester, using the exact parameters chosen for OOS. Run it in the in sample segment, and you'll see that any trade taken in the last IS day is taken in the first OOS day.
So there are two errors now:
1- The last day of IS is wrong in the WFA table (it shows it's ending one day earlier)
2- the last IS day is still used as the first OOS day
Therefore, the WFA analysis continues to suffer from look ahead bias and data snooping. This is a serious bug.
-
Votes +3
-
Project StrategyQuant X
-
Type Bug
-
Status Fixed
-
Priority Normal
History
jose moreira
22.12.2018 03:45Attachment Figure 1.png added
Attachment Figure 2.png added
Attachment Figure 3.png added
Attachment Figure 4.png added
Attachment Figure 5.png added
Mark Fric
28.12.2018 04:05Status changed from New to Fixed
The problem was that statistics for IS part was computed from IS date from + one more day. So the best optimization parameters were computed from one day longer period.
OOS was then computed from its correct period.
Without waiting to lessen it, I don't think it really affects WF results that much. The best parameters in IS part were computed from period that is 1 day longer and "sees" 1 day into the future (OOS part).
Big majority of generated the strategies wil be not affected by this at all.
jose moreira
28.12.2018 04:30I'm not sure how you and your team approach WFA/WFM, but look ahead bias of a single day may have severe effects on the development process, in particular when dealing with low-frequency trade strategies, where a trade taken on the first OOS day may affect the whole result of the OOS segment. This is a big deal because the "Randomly skip trades" function of the Monte Carlo engines do not work on WFA/WFM's OOS segments (only on "Main data"). In addition, when WFA/WFM OOS windows are short in duration (many developers use that), the effect of this single day bias is massive. Basically any strategy validated using a range of 5-to-10 days IS/OOS segments will be reported as highly profitable out of sample, all due to this one day bias. Of course this is a fine-tuning technical issue, but I see this is the level of excellence you are trying to achieve in SQX, so I stressed this point a lot when reporting the bug.
Anyways, thanks for fixing it. Looking forward to seeing it in Build 117.
SQX is becoming a great software. Unfortunately there are still too many bugs before prime time, but I see you are working hard on fixing them. So, congrats for that.
Best regards
Jose
© Copyright. All rights reserved. ProjectPanel.com
Would you please fix that?
Cheers
The figures contain explanations within them, but I summarize their content below:
Figure 1 - WFA results and IS dates (supposed to be)
Figure 2 - Retest of IS period using the same dates and parameters as in WFA (different results from WFA in Optimizer)
Figure 3 - Retest of IS period including the first OOS day in WFA (same results as IS in WFA in Optimizer)
Figure 4 - List of trades in IS retest (from Figure 4), including the first OOS day (note the last trade)
Figure 5 - Retest of first OOS segment (the first trade is the same as the last shown in Figure 4. That makes the OOS very profitable, but only because 12.01.2010 was IN SAMPLE, and therefore available for optimization). The problem repeats in all OOS segments with trades on the first OOS day.