The "Stability" calculations in the walk forward are incorrect - they are not being averaged by the number of days.

The "Stability" calculations in the walk forward are incorrect - they are not being averaged by the number of days.



WF Stability components allow you to compare performance between optimization and run (averaged by number of days, so it is not dependent on relative size of each period).


WF Net Profit Stability – Net profit performance in run vs in optimization part (in percent).


Value above 100% means that strategy performs better in run than in optimization part. 


Let’s say you specify condition WF Net Profit Stability > 60%.


This means that performance in run part (after optimization) should be at least 60% of performance in optimization period. So, for example, if the strategy made $1000 in optimization period, it should make at least $600 or more after optimization period to pass this condition.


This is important to evaluate because we want our strategy to perform well after we optimize it and this condition allows us to control this – in our condition we let the optimization pass only if the startegy performs at least at 60% of the optimized performance.



This means potentially all the stability calculations in walk forward are incorrect and the values are much lower than they should be. I have proved this for net profit so far.

Attachments
No attachments
  • Votes +5
  • Project StrategyQuant X
  • Type Bug
  • Status Fixed
  • Priority Normal

History

N
#1

nathan

21.11.2018 17:50

Task created

o
#2

Enric

21.11.2018 22:02
Voted for this task.
RL
#3

rickliao

22.11.2018 04:26
Voted for this task.
N
#4

nathan

22.11.2018 11:05
Voted for this task.
N
#5

nathan

23.11.2018 11:50
Sorry, but how is this build 116 and not 115 "urgent"?? 


This is a fundamental calculation error affecting all Stability calculations. All Stability Calculations and filters are wrong until this is fixed, therefore Walk Forward Analysis is currently useless.


We now need to wait 6 weeks until we can fully develop a strategy with SQX and use the software as intended/purchased?

MF
#6

Mark Fric

23.11.2018 13:13

Status changed from New to Refused

I looked at it, but we will be not changing the way it is computed. Most of the stats fields aren't dependent on number of trading days and thus shouldn't be normalized by days.


I don't appreciate your last comment. Just because stability is computed in a different way than you want it you say the whole Walk-Forward and whole SQ is useless.


There is a very simple way to fix it - simply don't use Net profit, but for example CAGR that is % based.


Or if you want to use Net profit normalized by number of days then you can use Avg. Profit per day.


There is also nothing preventing you from creating your own stats column.


N
#7

nathan

23.11.2018 15:07

"...Just because stability is computed in a different way than you want it..."


No, not how I want it, can't take credit for that I'm afraid, its how you wanted it: https://docs.strategyquant.com/walk-forward-optimization :)


I was just fishing around by trial and error in SQX for a way to compare WF IS metrics with WF. There is nothing in the manual. Stability sounded like a possibility, but again nothing about in the manual to confirm. Google turned up the above link, and I took a liking to it :)   .... So it will be computed differently in SQX. Now I know.


I can create my own stats columns too? - great. Now I know.


I think a lot of this frustration boils down to the fact there is scant detailed information in the manual and documentation for people like me who never used the old version and only recently came accross SQX for the first time. There is nothing on WF subset results (Result, Stability, Score) in the manual. I have queried these before, but still no explanation. I figured out what most of them do by trial and error, but I am still not 100% sure. Again, an explanation of all SQX functionality and options in the manual would really help. Half these things wouldn't come up if the manual was more complete - I don't appreciate wasting my time figuring what they do by experimentation, when a manual could explain it to me in 5 minutes.


So useless? No, I stand corrected now I know I can create my own stats columns for the Stability subset... But maybe give someone a gentle push to complete a more detailed manual.


MF
#8

Mark Fric

23.11.2018 15:19

Status changed from Refused to In progress

I'm aware that manual is still incomplete, we are working on it.


You are right that the description of Stability that you found  in old documentation doesn't match how it is computed in SQ X, i'll review it for the new build.


Until then you can still use Avg. profit per day, as I suggested.

N
#9

nathan

27.11.2018 15:14
Could you maybe share an example of how this is possible/created in SQX using Avg. Profit per Day or CAGR as you suggested above?


Assuming I should be using the WF Subset Stability data result in order to compare the WF run with the optimization part, both Avg. Days and CAGR return values of 2147483647%, which is not right.


If I select WF Result this give $80.72 which is correct. And selecting WF Score gives 115% (but I have no clue what 'score' refers to so can't judge this one).


I don't know exactly what WF Stability, Result and Score are doing/supposed to be doing as there is nothing in the manual or help in SQX. From  piecing bits of info together it looks the WF Stability data is comparing the WF Run Part with the optimization part, but using Avg. Profit per Day or CAGR returns unrealistically large values.


Any chance of an explanation?


Thanks!




.


N
#10

nathan

27.11.2018 22:20
Yearly Avg.Profit, Yearly Avg. Return. CAGR, Daily Avg. Profit, Monthly Avg, Profit for (WF Optimization, WF Stability) all give exactly the same value 2147483647%. This is true for any and every strategy.


"The number 2,147,483,647 (or hexadecimal 7FFF,FFFF16) is the maximum positive value for a 32-bit signed binary integer in computing. It is therefore the maximum value for variables declared as integers (e.g., as int) in many programming languages, and the maximum possible score, money, etc. for many video games. The appearance of the number often reflects an error, overflow condition, or missing value"


Wiki.


g
#11

geektrader

27.11.2018 22:32
Voted for this task.
h
#12

hankeys

28.11.2018 09:02
Voted for this task.
MF
#13

Mark Fric

28.11.2018 11:36

Status changed from In progress to Fixed

it is fixed. I added normalization by number of days for columns like Net profit, and refactored the whole algorithm.


Both Stability and Score now work correctly. I'll also write a short article explaining them.



Votes: +5

Drop files to upload

or

choose files

Max size: 5MB

Not allowed: exe, msi, application, reg, php, js, htaccess, htpasswd, gitignore

...
Wait please