Major failure in Walk Forward Engine.

Hi, 

I’ve detected two major failures in the Walk Forward engines of StrategyQuantX. In my view it nearly invalidates any WF analysis in the software. Images attached

Failure 1 - When running Walk Forward optimization or matrix, the software considers the last “In Sample-IS” day as the first “Out of Sample-OOS” day. See image below. This is a major mistake and results in the classic “data snooping”. The in sample period is looking one day into the future (OOS) and then the engine considers that the last IS day (07.01.2010 in the image) is ALSO out of sample, when it is not (because it was used for IS optimization). First I thought that Strategy Quant behaved similarly to Metatrater and ignored the last day of the range, but after checking many IS and OOS ranges, I realized that actually it uses the last IS day also as the first OOS day. 

In real life, an in sample period ending on 07.01.2010 would have an OOS starting on the 08.01.2010. This is a very serious error in the software. 

Failure 2 - The walk forward engine also uses the last OOS day as the first day for the next OOS segment. See image below. An OOS segment ending on 12.01.2010 would have the next OOS starting on 13.01.2010  but the software duplicates the trading session of 12.01.2010. This means that any trade taken in the last OOS day will happen again in the next OOS segment, and completely destroy the WF analysis. See duplicate trades in attached image

As a result of these error, the software not only creates look ahead bias, but also suffers from data snooping and creates invalid list of trades for OOS segments. As an example of what that could cause, you can see below that absolutely any strategy with few trades will have an excellent performance out of sample on walk forward analysis, simply because the last trade in sample occurs as the first trade out of sample. 


Would you look into this?

Best regards

Jose
Attachments
No attachments
  • Votes +6
  • Project StrategyQuant X
  • Type Bug
  • Status Fixed
  • Priority Normal

History

m
#1

jose moreira

18.12.2018 16:53

Task created

m
#2

jose moreira

18.12.2018 16:57

Description changed:

Hi, 

I’ve detected two major failures in the Walk Forward engines of StrategyQuantX. In my view it nearly invalidates any WF analysis in the software. Images attached

Failure 1 - When running Walk Forward optimization or matrix, the software considers the last “In Sample-IS” day as the first “Out of Sample-OOS” day. See image below. This is a major mistake and results in the classic “data snooping”. The in sample period is looking one day into the future (OOS) and then the engine considers that the last IS day (07.01.2010 in the image) is ALSO out of sample, when it is not (because it was used for IS optimization). First I thought that Strategy Quant behaved similarly to Metatrater and ignored the last day of the range, but after checking many IS and OOS ranges, I realized that actually it uses the last IS day also as the first OOS day. 

In real life, an in sample period ending on 07.01.2010 would have an OOS starting on the 08.01.2010. This is a very serious error in the software. 

Failure 2 - The walk forward engine also uses the last OOS day as the first day for the next OOS segment. See image below. An OOS segment ending on 12.01.2010 would have the next OOS starting on 13.01.2010  but the software duplicates the trading session of 12.01.2010. This means that any trade taken in the last OOS day will happen again in the next OOS segment, and completely destroy the WF analysis. See duplicate trades in attached image

As a result of these error, the software not only creates look ahead bias, but also suffers from data snooping and creates invalid list of trades for OOS segments. As an example of what that could cause, you can see below that absolutely any strategy with few trades will have an excellent performance out of sample on walk forward analysis, simply because the last trade in sample occurs as the first trade out of sample. 


Would you look into this?

Best regards

Jose

Attachment Duplicate trades on ISOOS edges.png added

Attachment Final result.png added

Attachment Failure 1.png added

Attachment Failure 2.png added

g
#3

geektrader

18.12.2018 18:00
Voted for this task.
a
#4

Ash24FX

18.12.2018 18:24
Voted for this task.
mp
#5

Michele

18.12.2018 18:51
Voted for this task.
o
#6

Enric

18.12.2018 21:19
Voted for this task.
m
#7

mabi

19.12.2018 23:42
Voted for this task.
MF
#8

Mark Fric

20.12.2018 20:46

Status changed from New to Fixed

you were right, both things should be fixed now


m
#9

jose moreira

20.12.2018 21:00
Great Mark. Is the fixed WFA/WFM available already or only when the new build is released? If it's available now, what should we do to have the fixed solution (just restart SQX) ?
N
#10

nathan

22.12.2018 06:31
Voted for this task.
MF
#11

Mark Fric

26.12.2018 13:52
it was supposed to be fixed, but it was reported again, so I'll look at it again. Otverwse you need only to update to the latest version and restart SQ.
MF
#12

Mark Fric

28.12.2018 13:45
it is fixed

Votes: +6

Drop files to upload

or

choose files

Max size: 5MB

Not allowed: exe, msi, application, reg, php, js, htaccess, htpasswd, gitignore

...
Wait please