Build (Task) : Full Settings : Strategy Quality Fitness on In Sample Period or full period
I don't see if the quality Fitness is calculated on the In Sample Period or the full period of the task.
1/ If the calculus of the quality Fitness is made on the full period data, is it possible to make the calculus on the IS period only as an option ?
If an Out of Sample Period is present, and the Quality Fitness is calculated on both period IS and OOS, the fitness ranking is importing in databank more strategies with the good OOS dismissing the strategies with bad OOS for the same parameter.
This is not good. This is inducing trader in error
SQX would bring a lot more good strategies OOS than it should. This is bringing Statistical error.
This mean that there no real Out Of Sample period when you are building your strategies with a Strategy Quality Fitness based on all the period Data.
If you are searching strategies with this process, the Strategy Quality Fitness based on all the period Data will import in your databank strategies with good results In Sample and Out Of Sample,
you will find that your parameters IS are good to filter your strategies, as there is more good OOS results in you Databanks.
However, as soon as you will search strategies with the same parameters for the next period without OOS , the Strategy Quality Fitness will be based only on IS period Data, will give all the strategies without filtering them as they will no OOS period, and your results will be much worse , of course.
2/ Furthermore, in the databank column, the fitness IS and the fitness OOS have the same results, this is bringing false positive results as well. if people select fitness as a filter as they don't realize they are selecting the OOS fitness period as well
We should have a different results on the IS fitness and the OOS Fitness.
-
Votes +3
-
Project StrategyQuant X
-
Type Feature
-
Status New
-
Priority Normal
History
Emmanuel
29.12.2021 22:26Description changed:
I don't see if the quality Fitness is calculated on the In Sample Period or the full period of the task.
1/ If the calculus is made on the full period data, is it possible to make the calculus on the IS period only as an option ?
If an Out of Sample Period is present, and the Quality Fitness is calculated on both period IS and OOS, the fitness ranking is importing in databank more strategies with the good OOS dismissing the strategies with bad OOS for the same parameter.
This is not good. This is inducing trader in error
SQX could bring a lot more good strategies OOS than it should. This is bringing Statistical error.
2/ Furthermore, in the databank column, the fitness IS and the fitness OOS have the same results, this is bringing false positive results as well. if people select fitness as a filter as they don't realize they are selecting the OOS fitness period as well
We should have a different results on the IS fitness and the OOS Fitness.
Emmanuel
29.12.2021 23:45Description changed:
I don't see if the quality Fitness is calculated on the In Sample Period or the full period of the task.
1/ If the calculus of the quality Fitness is made on the full period data, is it possible to make the calculus on the IS period only as an option ?
If an Out of Sample Period is present, and the Quality Fitness is calculated on both period IS and OOS, the fitness ranking is importing in databank more strategies with the good OOS dismissing the strategies with bad OOS for the same parameter.
This is not good. This is inducing trader in error
SQX would bring a lot more good strategies OOS than it should. This is bringing Statistical error.
This mean that there no real Out Of Sample period when you are building your strategies with a Strategy Quality Fitness based on all the period Data.
If you are searching strategies with this process, the Strategy Quality Fitness based on all the period Data will import in your databank strategies with good results In Sample and Out Of Sample,
you will find that your parameters IS are good to filter your strategies, as there is more good OOS results in you Databanks.
However, as soon as you will search strategies with the same parameters for the next period without OOS , the Strategy Quality Fitness will be based only on IS period Data, will give all the strategies without filtering them as they will no OOS period, and your results will be much worse , of course.
2/ Furthermore, in the databank column, the fitness IS and the fitness OOS have the same results, this is bringing false positive results as well. if people select fitness as a filter as they don't realize they are selecting the OOS fitness period as well
We should have a different results on the IS fitness and the OOS Fitness.
Emmanuel
03.01.2022 22:01If you add an Out of Sample period to your build task, the task is unable to search on the In Sample period only.
If the fitness is based on all the period, it means that you have to build the strategies in one task , and retest you out of sample on a second task.
I tested it, there is a big difference, if I am searching the in sample in on task and retest the Out of sample in the second task
Emmanuel
04.01.2022 15:351/ select fitness on NetProfit, start build
2/ in the In Sample, select the worse net Profit
3/ Look at the OOS Net profit, it would be extremely profitable Net profit
Why ? this is simple :
If the fitness is based on the Full Net Profit : (Instead of IS)
Full period Net Profit = IS Net Profit + OOS Net Profit
if you select the worse IS Net profit , in a group of strategies with good Full period Net Profit , you will get only excellent OOS Net Profit.
This is mathematical with the formula above
Of course, this won't work in reality, if you select the same parameters in the IS period only without OOS to trade in reality , your results would be completely different
One way to go around this problem for the moment is to create Netprofit column filtered IS and to select it in the "Weighted Fitness" to make sure to use the IS Stats only
Emmanuel
04.01.2022 15:58How do we select the Fitness on the Walk Forward test ? and other cross check test ?
It is possible to check the fitness of the cross check test too ?
bentra
14.01.2022 06:04EDIT:
Apologies, you're right, the fitness ranking used for the databank population control does not have an option to select IS. I guess you need to build with no OOS to be sure. Is that a big deal? You will actually save CPU. Or just stop when bank is full and move on to the next step is what I do.
EDIT:
About the fitness IS and fitness OS, you're right they are the same it is bugged, I voted your task on that one specifically.
© Copyright. All rights reserved. ProjectPanel.com
