biases caused by genetic optimizer, please give an option to turn it off and either do exhaustive or just make random optimization similar to MonteCarlo
-In the WF there is a potential look-ahead bias:
sim/sim mode will use the whole sample at once for an initial optimization. The genetics kicks in on the whole sample causing more iterations to converge around the higher fitness iterations on the whole sample. So when the best settings are chosen for a specific segment it is more likely to choose one that works well OOS as there will be more of those to choose from. This could be a small look-ahead bias but it exists and is not good.
-In the SPP study there is a similar bias:
The genetics kick in around the higher fitness and so we have more of the higher fitness iterations than we would if it was just random or exhaustive so stats such as % profitable iterations etc will be overrepresented.
If we could turn off genetics for the optimizer then the distribution could be exhaustion (run every single possibility) or random (similar to Monte-Carlo) instead of skewed towards what works well on the whole sample so that would be nice to alleviate these biases. Even if you finally add the exact/exact function (EDIT: it appears SQX team finally added exact/exact) of WF, this feature would still be nice so we can run sim/sim without bias and save cpu.
EDIT: can we also have this option for Seq Opt and all forms of optimizations too if needed?
EDIT: It just occurred to me that some people will want to run optimizations or SPP to absolute exhaustion. So, when turning off genetics, could we please choose between attempting to run every single combination once (exhaustion) as well as being able to make it jump around randomly like monte-carlo? Or perhaps it could even be automatic for instance if max optimizations is set for 1000 and genetics is turned off and the total possible combos are only 500 then of course just do an exhaustive search but if total possible combos are 2000 and maz optimizations are set for 1000 then use montecarlo style.
On a side note, It turns out that even searching a 4x4 area with only 16 combos in an SPP will give a different random result every time. Running a search to exhaustion means we can reproduce the same results instead of being random every time.
-
Votes +16
-
Project StrategyQuant X
-
Type Feature
-
Status In progress
-
Priority High
History
bentra
06.12.2022 21:47Description changed:
sim/sim mode will use the whole sample at once for an initial optimization. The genetics kicks in on the whole sample causing more iterations to converge around the higher fitness iterations on the whole sample. So when the best settings are chosen for a specific segment it is more likely to choose one that works well OOS as there will be more of those to choose from. This could be a small look-ahead bias but it exists and is not good.
In the SPP study there is a similar bias:
The genetics kick in around the higher fitness and so we have more of the higher fitness iterations than we would if it was just random or exhaustive so stats such as % profitable iterations etc will be overrepresented.
If we could turn off genetics for the optimizer then the distribution would be random similar to Monte-Carlo instead of skewed towards what works well on the whole sample so that would be nice to alleviate these biases. Even if you finally add the exact/exact function (EDIT: it appears SQX team finally added exact/exact) of WF, this feature would still be nice so we can run sim/sim without bias and save cpu.
EDIT: can we also have this option for Seq Opt too if needed?
EDIT: It just occurred to me that some people will want to run large (1000+) optimizations to absolute exhaustion. So, when turning off genetics, could we please choose between attempting to run every single combination once (exhaustion) as well as being able to make it jump around randomly like montecarlo? Or perhaps it could even be automatic for instance if max optimizations is set for 1000 and genetics is turned off and the total possible combos are only 500 then of course just do an exhaustive search but if total possible combos are 2000 then use montecarlo style.
bentra
25.12.2022 00:32Mark Fric
01.03.2023 11:36Status changed from New to In progress
Priority changed from Normal to High
You are able to select max optimizations limit to up to 10.000.000 for all WF and optimization crosschecks, as well as in Optimizer. This is the way how to "turn" genetics off.
If the max optimizations limit is set to 10.000.000 it will use brute force if the total number of combinations is smaller than this.
But 10.000.000 optimizations is extreme number, if your strategy has that many combinations you'd wait for days to finish optimization of one such strategy using brute force.
If your strategy has "normal" number of parameters and optimizations then this is the way how to make it use brute force.
> It turns out that even searching a 4x4 area with only 16 combos in an SPP will give a different random result every time so turning off genetics is preferable here so we can reproduce the test exactly as before > by doing an exhaustive search. There is no reason not to try all the combos if there are only 16 possibilities!
you didn't say how did you configure Max optimizations limit. It is genetics, so the results could differ, but the bigger the limit you set the more it should converge to the "correct" numbers.
Anyway, we'll increase limit of max. optimizations to 1.000.000.000, but I don't know who would really use it.
I think the key is in producing strategies with as small level of complexity as possible, which means also small number of params to optimize.
bentra
16.04.2023 04:59It would be nice though if we could actually turn off genetics and use a monte-carlo (random) engine instead of genetics. Running to exhaustion is too much for me yet I do not want the bias of the genetic engine during SPP or during a "simulated/simulated" optimization. As per the original request of this ticket.)
>you didn't say how did you configure Max optimizations limit. It is genetics, so the results could differ, but the bigger the limit you set the more it should converge to the "correct" numbers.
I meant total optimizations possible was only 16 so the lowest selectable option of 100 still easily covers all the optimizations possible, in this case the old genetics were making the optimization impossible to reproduce consistently even though there were only 16 combos. Looks like you fixed that part now so all good.
© Copyright. All rights reserved. ProjectPanel.com
