[Build 113] Stop including the in-sample portion of walk forwards in...

Please stop including the in sample portion of walk forwards in stats such as “ret/DD ratio (walk-forward optimization, WF result.)” Including the in-sample portions in walk-forward stats as Strategyquant does seems awfully illogical. Any good trading book discussing walk forwards that I have read only analyzes the out of sample portions.
Attachments
No attachments
  • Votes +6
  • Project StrategyQuant X
  • Type Feature
  • Status Fixed
  • Priority Normal

History

b
#1

bentra

28.10.2018 12:39

Task created

b
#2

bentra

28.10.2018 12:47
Voted for this task.
N
#3

nathan

28.10.2018 18:52
Voted for this task.
b
#4

bentra

29.10.2018 10:32
Really no need for it in the equity charts either.
MF
#5

Mark Fric

29.10.2018 14:18

Status changed from New to Waiting for information

I'm not sure we should do it like this. We don't use In sample parts in WF stats, except the very first part. 

The first training part is quite big and if we'll not use it then the net profit of the strategy after walk-forward will be much smaller than original strategy.


Every platform I know displays it the same way also in the equity chart - te first training part is there, and then the run parts of the reoptimizations.


b
#6

bentra

29.10.2018 15:30
Why does the size of the net profit matter? They will all be smaller observed samples and have smaller net profits and still be comparable to each other and could be compared to the same sample of the origina run too (without the first part). You could compare separately the first part of the original to the first part of the in sample optimization part too on another chart (without having to combine in samp with oos) 


The purpose of a walk forward is to analyze how we do AFTER the optimization, on a blind sample. When you include the optimization you're defeating the purpose. I will ref "Building Winning Algorithmic Trading Systems" by Kevin Davey (award winning full time trader, with over 25 years trading experience). You can also check out how this software keeps in sample and oos seperate like it should: see WFP works here: https://www.tradelikeamachine.com/backtesting-software-walk-forward-pro/hub/walk-forward-pro-videos


I'd like to be able to compare PF, ret/dd etc of one out of sample walk forward to another. I suppose leaving it in the chart doesn't hurt that much but I would prefer to see the out of sample portion begin at the pl 0 line. Please consider seeing how many people vote and maybe giving us some options. Maybe, at least a new stat such as ret/DD ratio (walk-forward optimization, WF result - blind data only.) and an option to cut out in sample data from the chart.

It's very annoying to see a stat like profit factor show 1.35 for the walk forward optimization yet none of the walk forward out of sample segments were profitable. I just want to know about the oos parts.

All the tests this software can do are insanely awesome. There's just this one little thing that could make it slightly better. =)

m
#7

mabi

29.10.2018 22:59
There is another task here which I cant find now for some reason that would take care of this aimed for B114. SQX will be able to save all walk forward periods to the strategy. When the walkforward periods settings is saved in the strategy it can then be re-tested and analyzed using montecarlo as well. I ask for this option in the forum already years ago at serval occasions and it seems it will now be implemented. This way We can even montecarlo the result and compare different periods from Walk forward matrix
m
#8

mabi

29.10.2018 23:02
Voted for this task.
mp
#9

Michele

30.10.2018 00:10
Voted for this task.
b
#10

bentra

30.10.2018 00:21

Subject changed from [Build 113] Stop including the in-sample portion of walk forwards in... to Please stop including an in-sample portion of the walk forward tests in various WF analysis, stats and charts... OOS only please! (Or some options to analyze isolated OOS)

mp
#11

Michele

30.10.2018 00:26

Subject changed from Please stop including an in-sample portion of the walk forward tests in various WF analysis, stats and charts... OOS only please! (Or some options to analyze isolated OOS) to [Build 113] Stop including the in-sample portion of walk forwards in...

I agree with this task. It is important the result after the optimization, the in sample initial period is overoptimized and distorts the results. 


If you cut the in Sample period you'll see that the optimized out of sample equity is the same or only very little different from the original equity, because it seems the Walk Forward process doesn't work.  

I explain better the problem in this task: https://roadmap.strategyquant.com/tasks/sq4_3274


Michele

b
#12

bentra

02.11.2018 06:53
Yes you're right Michel, the process needs to work too and I have voted for your task.

b
#13

bentra

04.11.2018 08:44
another alternative solution:
Just have the oos button in the results for a walkforward only show stats from oos. Currently it shows nothing at all.
b
#14

bentra

07.11.2018 04:58
I'm reading "Building Winning Algorithmic Trading Systems: A Trader's Journey From Data Mining to Monte Carlo Simulation to Live Trading (Wiley Trading)" by Kevin Davey and wanted to share this quote with you.

"The idea behind walk-forward analysis is simple: the performance results and the optimized results are based on two different data sets . This can be seen in Figure 5.5 . Walk-forward analysis is simply the aggregate of many out-of-sample periods, stitched together."

Start reading this book for free: http://a.co/67r1oEk

N
#15

nathan

07.11.2018 09:20
I agree with this. Unless you are analysing the OOS components of the Walk Forward only and base the stats on the concatenated results of the walk forward OSS's then, simply put, it is not a walk forward analysis. WFA, in the final analysis, done properly, is pretty much an 'idiot proof way of avoiding curve fit' (quoting Pardo). If it doesn't pass WFA, then either the strategy is BS or you made a mistake in the optimization/development process. 


This software is absolutely awesome, dont get me wrong, but you have to know what you are doing with regards to systems development and statistics. If you blindly follow all the default settings and filters without questioning each and everyone of them, you are going to be left with some false positives at the end of the robustness funnel, in my humble opinion at least.


@bentra - I will checkout the Davey book, thanks for the tip. The two books on my desk at the moment are 'Evaluation and optimization of trading strategies - PARDO' and 'Trading Systems - Jaekle & Tomasini'

MF
#16

Mark Fric

07.11.2018 16:30

Status changed from Waiting for information to Fixed

done. We fixed a bug switching between Full / IS / OOS parts in equity chart and results, so now you can display the part you want. OOS part is the default.
k
#17

Karish

14.11.2018 14:24
Voted for this task.
o
#18

Enric

30.12.2018 13:01
Voted for this task.

Votes: +6

Drop files to upload

or

choose files

Max size: 5MB

Not allowed: exe, msi, application, reg, php, js, htaccess, htpasswd, gitignore

...
Wait please