The "Stability" calculations in the walk forward are incorrect - they are not being averaged by the number of days.
WF Stability components allow you to compare performance between optimization and run (averaged by number of days, so it is not dependent on relative size of each period).
WF Net Profit Stability – Net profit performance in run vs in optimization part (in percent).
Value above 100% means that strategy performs better in run than in optimization part.
Let’s say you specify condition WF Net Profit Stability > 60%.
This means that performance in run part (after optimization) should be at least 60% of performance in optimization period. So, for example, if the strategy made $1000 in optimization period, it should make at least $600 or more after optimization period to pass this condition.
This is important to evaluate because we want our strategy to perform well after we optimize it and this condition allows us to control this – in our condition we let the optimization pass only if the startegy performs at least at 60% of the optimized performance.
This means potentially all the stability calculations in walk forward are incorrect and the values are much lower than they should be. I have proved this for net profit so far.
This is a fundamental calculation error affecting all Stability calculations. All Stability Calculations and filters are wrong until this is fixed, therefore Walk Forward Analysis is currently useless.
We now need to wait 6 weeks until we can fully develop a strategy with SQX and use the software as intended/purchased?
Status changed from New to Refused
I don't appreciate your last comment. Just because stability is computed in a different way than you want it you say the whole Walk-Forward and whole SQ is useless.
There is a very simple way to fix it - simply don't use Net profit, but for example CAGR that is % based.
Or if you want to use Net profit normalized by number of days then you can use Avg. Profit per day.
There is also nothing preventing you from creating your own stats column.
"...Just because stability is computed in a different way than you want it..."
No, not how I want it, can't take credit for that I'm afraid, its how you wanted it: https://docs.strategyquant.com/walk-forward-optimization :)
I was just fishing around by trial and error in SQX for a way to compare WF IS metrics with WF. There is nothing in the manual. Stability sounded like a possibility, but again nothing about in the manual to confirm. Google turned up the above link, and I took a liking to it :) .... So it will be computed differently in SQX. Now I know.
I can create my own stats columns too? - great. Now I know.
I think a lot of this frustration boils down to the fact there is scant detailed information in the manual and documentation for people like me who never used the old version and only recently came accross SQX for the first time. There is nothing on WF subset results (Result, Stability, Score) in the manual. I have queried these before, but still no explanation. I figured out what most of them do by trial and error, but I am still not 100% sure. Again, an explanation of all SQX functionality and options in the manual would really help. Half these things wouldn't come up if the manual was more complete - I don't appreciate wasting my time figuring what they do by experimentation, when a manual could explain it to me in 5 minutes.
So useless? No, I stand corrected now I know I can create my own stats columns for the Stability subset... But maybe give someone a gentle push to complete a more detailed manual.
Status changed from Refused to In progress
You are right that the description of Stability that you found in old documentation doesn't match how it is computed in SQ X, i'll review it for the new build.
Until then you can still use Avg. profit per day, as I suggested.
Assuming I should be using the WF Subset Stability data result in order to compare the WF run with the optimization part, both Avg. Days and CAGR return values of 2147483647%, which is not right.
If I select WF Result this give $80.72 which is correct. And selecting WF Score gives 115% (but I have no clue what 'score' refers to so can't judge this one).
I don't know exactly what WF Stability, Result and Score are doing/supposed to be doing as there is nothing in the manual or help in SQX. From piecing bits of info together it looks the WF Stability data is comparing the WF Run Part with the optimization part, but using Avg. Profit per Day or CAGR returns unrealistically large values.
Any chance of an explanation?
Thanks!
.
"The number 2,147,483,647 (or hexadecimal 7FFF,FFFF16) is the maximum positive value for a 32-bit signed binary integer in computing. It is therefore the maximum value for variables declared as integers (e.g., as int) in many programming languages, and the maximum possible score, money, etc. for many video games. The appearance of the number often reflects an error, overflow condition, or missing value"
Wiki.
Status changed from In progress to Fixed
Both Stability and Score now work correctly. I'll also write a short article explaining them.