strange behavior of genetics

if i run genetics on 5 years of data - no IS/OOS sampling - so everything is only IS my project is running 150k strats per hour



but if i do this simply change - add OOS before the IS - so the IS remain totally same 5 yers and start again this project without doing anything else - the project will slow down to only 40k strats per hour


this behavior is not logical, because genetics should look only to IS and amount of data is the same, so the speed should be the same, but its not


its seems, that genetics is using somehow whole data sample IS+OOS

Attachments
1.jpg
(68.49 KiB)
2.jpg
(99.03 KiB)
  • Votes +11
  • Project StrategyQuant X
  • Type Bug
  • Status Refused
  • Priority Normal

History

h
#1

hankeys

22.12.2020 09:51

Task created

h
#2

hankeys

22.12.2020 09:51
Voted for this task.
b
#3

bentra

22.12.2020 10:18
I guess it runs tests on all data in the moment of testing which I guess is fine in case you have an oos filter. If you do not have an oos filter you can turn off oos....

I imagine you will get the same results if you do that test with random generator too.... or really with any task. The test runs first then filters are checked after so it needs to run a full test first in case you have an oos filter.
h
#4

hankeys

22.12.2020 10:32
this behavior is not acceptable - the slowdown occurs right after the start button is clicked, so the islands are starting to fill with initial population, which cant be filtered by any OOS data. Initial population must be choosen only from IS data


after that genetics will start and make the filtering by IS or OOS ranking - this has nothing to do whats happening in b130


so by my opinion added OOS if IS stays the same cant result to slow down the building of initial population



b
#5

bentra

22.12.2020 10:52

Attachment Screenshot 135.png added

Screenshot 135.png
(154.24 KiB)
"so the islands are starting to fill with initial population, which cant be filtered by any OOS data. Initial"

Yes actually init pop can currently be filtered by OOS data. Would anyone ever WANT to? probably not. I don't know why this is possible but it is.

It should probably be ISV not OOS here but sadly it is really OOS. According to my testing ISV is included in the IS part so there's no way to separate IST and ISV at the moment in the init pop filters <--- this is a problem.

To me it seems most logical to not allow OOS in builder data, instead only allow IST and ISV. Then also make the init pop filters able to filter by those. It would be less confusing and more useful.

In the meantime can't you just run builder without OOS then do a retest after which includes the oos?


b
#6

bentra

22.12.2020 11:10
Voted for this task.
b
#7

bentra

22.12.2020 11:16
In any case, build task and retest task does only one whole main data test first for each strategy then checks filters after. (EDIT: It's true that checks against the global ranking filters are not done until the islands are full BUT when it checks the final global ranking filters it just uses the test from memory that's already been done during building, I'm 99% sure.)

 It does not do two separate tests I.E. test in sample in one test then check filters then do OOS test if passes filters (You can accomplish this though by using two separate tasks)
IH
#8

clonex / Ivan Hudec

23.12.2020 21:54
Voted for this task.
g
#9

Gin

24.12.2020 11:42
Voted for this task.
o
#10

Enric

24.12.2020 20:20
Voted for this task.
b
#11

bentra

27.12.2020 07:40
The more I think about it the more I agree with Hankeys. It would be nice to check in-sample filters before proceeding to do an OOS test.
g
#12

gcraenen

27.12.2020 12:38
Voted for this task.
MF
#13

Mark Fric

28.01.2021 14:10

Subject changed from CRUCIAL - strange behavior of genetics to strange behavior of genetics

g
#14

geektrader

28.01.2021 14:41
Voted for this task.
DR
#15

mentaledge

28.01.2021 16:43
Voted for this task.
k
#16

Karish

30.01.2021 15:08
Voted for this task.
MF
#17

Mark Fric

11.02.2021 09:24

Status changed from New to Refused

the explanation is simple - you are backtesting the strategy on 4 years in faster case and 16 years in slower case.


Strategy is always backtested on full configured data range, stats for IS / OOS parts are computed only after that.


I understand your point that to speed it up it could backtest strategy only on IS part, but it is not that simple. 

You can have multiple IS parts scattered in the range, and you can have conditions anywhere that count with Full sample or OOS.

JJ
#18

jjsb41

25.01.2022 23:36
Voted for this task.

Votes: +11

Drop files to upload

or

choose files

Max size: 5MB

Not allowed: exe, msi, application, reg, php, js, htaccess, htpasswd, gitignore

...
Wait please