<- Back to Tasks

[Build 137 RC2] Contamination OOS statistics from IS statistics




As you can see on the "Capture end test 2", the In Sample period is from 2019.5.01 to 2020.05.16 and the OOS period is from 2020.05.17 to 2020.05.23



However, you can see in the list of trades, (on the "Capture end test"), the IS last trade is closing only  the 2020.05.19 in the OOS period with a standard "Exit signal" .


Furthermore, the first OOS trade start only after the end of the last IS trade. Instead it should start from the 17 or 18 of may 2020.


The consequence of closing the last trade outside of the IS period and inside of the OOS period is that there are  error of statistics calculus as we are using the OOS period to finish the IS trade.


Furthermore, as we are using OOS period to finish the IS period, the OOS trade period start later after the last IS trade. (the OOS kpi is less accurate)


Statistics Contamination is very serious because it is giving Kpi errors, this is why I ask to have an exit "EndTest" at each end In Sample period or Out Of Sample period to close any open trade.



In Build 137 RC1-2 , we have the same behavior : the previous In sample period will contaminate the next In Sample Validation period as well :

The last trade of the last In Sample Period will close in the next In Sample Validation period.

This is making the In sample Validation statistics not reliable as well if the last trade of the last ISV period is closing in the next ISV period.



Attachments
Capture end test.jpg
Capture end test.jpg
(76.83 KiB)
Capture end test 2.jpg
Capture end test 2.jpg
(83.52 KiB)
  • Votes +7
  • Project StrategyQuant X
  • Type Feature
  • Status New
  • Priority Normal

History

E
#1

Emmanuel

24.11.2021 21:59

Task created

E
#2

Emmanuel

24.11.2021 21:59
Voted for this task.
b
#3

bentra

26.11.2021 18:01
Voted for this task.
AS
#4

alisenotrade

30.11.2021 09:53

You are using the stop system and it is normal for a buyStop order to be placed in IS and the order to be converted to Market in OOS

IS and OOS are just statistical intervals, and at the end of IS, the orders are not going to be removed and wait for the signal again in OOS.

o
#5

Enric

31.01.2022 18:51
Voted for this task.
E
#6

Emmanuel

26.03.2022 15:57
. 
b
#7

bentra

16.04.2022 16:02
Some tests where holding times are very long will be more affected by IS trades going deep into OOS. But on the other hand, some tests where holding times are shorter and tests with many tiny OOS periods could be affected more by too many early termination of trades due to "EndTest." A third option could be simply omitting trades completely that overlap IS and OOS periods but this would reduce statistical significance. 

If this functionality is modified it should be optional, please.


E
#8

Emmanuel

16.04.2022 16:26
I agree, we need it as an option to keep flexibility


I see 3 options :


- We let the trade as it is even if it enter in OOS. This is terrible in my experience. Good strategies may look interesting and may be chosen instead of more robust one.  I had bad OOS results in the past with this solution.

- We disregard the trade closing in OOS period. this could be a simpler solution

- We close the trade at the end of IS period. Like we do on Friday as an option


Thank you Bentra for your comment and your vote :)



IN
#9

altenburg90

18.04.2022 02:03
Voted for this task.
HH
#10

Hans

21.10.2022 01:00
Voted for this task.
TT
#11

Tamas

11.11.2022 14:33

Type changed from Bug to Feature

Status changed from In progress to New

Ok, we will add a special setting for that "Dismiss trades that start in IS period and end in OOS period"


E
#12

Emmanuel

11.11.2022 17:08
this will be an excellent safety

Thank you

k
#13

Karish

15.11.2022 20:49
Voted for this task.
HF
#14

Vence_Jo

15.11.2022 21:04
Voted for this task.
E
#15

Emmanuel

30.04.2023 14:30

Subject changed from Contamination OOS statistics from IS statistics to [Build 137 RC2] Contamination OOS statistics from IS statistics

E
#16

Emmanuel

30.04.2023 14:32
We still have this bug, with Build 137 RC1-2


The trade was closed in the next period ISV1


The In sample statistics are not reliable if the last trade is closing in the next period.


We need it as soon as possible



E
#17

Emmanuel

30.04.2023 14:45

Description changed:




As you can see on the "Capture end test 2", the In Sample period is from 2019.5.01 to 2020.05.16 and the OOS period is from 2020.05.17 to 2020.05.23



However, you can see in the list of trades, (on the "Capture end test"), the IS last trade is closing only  the 2020.05.19 in the OOS period with a standard "Exit signal" .


Furthermore, the first OOS trade start only after the end of the last IS trade. Instead it should start from the 17 or 18 of may 2020.


The consequence of closing the last trade outside of the IS period and inside of the OOS period is that there are  error of statistics calculus as we are using the OOS period to finish the IS trade.


Furthermore, as we are using OOS period to finish the IS period, the OOS trade period start later after the last IS trade. (the OOS kpi is less accurate)


Statistics Contamination is very serious because it is giving Kpi errors, this is why I ask to have an exit "EndTest" at each end In Sample period or Out Of Sample period to close any open trade.



In Build 137 RC1-2 , we have the same behavior : the previous In sample period will contaminate the next In Sample Validation period as well :


The last trade of the last In Sample Period will close in the next In Sample Validation period.





E
#18

Emmanuel

30.04.2023 14:46

Description changed:




As you can see on the "Capture end test 2", the In Sample period is from 2019.5.01 to 2020.05.16 and the OOS period is from 2020.05.17 to 2020.05.23



However, you can see in the list of trades, (on the "Capture end test"), the IS last trade is closing only  the 2020.05.19 in the OOS period with a standard "Exit signal" .


Furthermore, the first OOS trade start only after the end of the last IS trade. Instead it should start from the 17 or 18 of may 2020.


The consequence of closing the last trade outside of the IS period and inside of the OOS period is that there are  error of statistics calculus as we are using the OOS period to finish the IS trade.


Furthermore, as we are using OOS period to finish the IS period, the OOS trade period start later after the last IS trade. (the OOS kpi is less accurate)


Statistics Contamination is very serious because it is giving Kpi errors, this is why I ask to have an exit "EndTest" at each end In Sample period or Out Of Sample period to close any open trade.



In Build 137 RC1-2 , we have the same behavior : the previous In sample period will contaminate the next In Sample Validation period as well :

The last trade of the last In Sample Period will close in the next In Sample Validation period.

This is making the In sample Validation statistics not reliable if the last trade is closing in the next period.



E
#19

Emmanuel

30.04.2023 14:47

Description changed:




As you can see on the "Capture end test 2", the In Sample period is from 2019.5.01 to 2020.05.16 and the OOS period is from 2020.05.17 to 2020.05.23



However, you can see in the list of trades, (on the "Capture end test"), the IS last trade is closing only  the 2020.05.19 in the OOS period with a standard "Exit signal" .


Furthermore, the first OOS trade start only after the end of the last IS trade. Instead it should start from the 17 or 18 of may 2020.


The consequence of closing the last trade outside of the IS period and inside of the OOS period is that there are  error of statistics calculus as we are using the OOS period to finish the IS trade.


Furthermore, as we are using OOS period to finish the IS period, the OOS trade period start later after the last IS trade. (the OOS kpi is less accurate)


Statistics Contamination is very serious because it is giving Kpi errors, this is why I ask to have an exit "EndTest" at each end In Sample period or Out Of Sample period to close any open trade.



In Build 137 RC1-2 , we have the same behavior : the previous In sample period will contaminate the next In Sample Validation period as well :

The last trade of the last In Sample Period will close in the next In Sample Validation period.

This is making the In sample Validation statistics not reliable as well if the last trade of the last ISV period is closing in the next ISV period.



TT
#20

Tamas

02.05.2023 08:48
Hello Emmanuel, we plan to do it in next build 138.


SQX 137 is almost ready, we are fine-tuning the last details. 

E
#21

Emmanuel

05.11.2023 02:01

Attachment image-0.png added

Attachment image-1.png added

image-0.png
image-0.png
(26.27 KiB)
image-1.png
image-1.png
(74.64 KiB)
Here is another example of IS OOS contamination :


1/   The Out of Sample period start on the 2015 12 29 (image 0)


And the last trade In Sample is closing on the 2016 1 22, at the end of the OOS period  (image 1)


2/  Furthermore, as the last IS trade closed much later, we have very few OOS trades


3/  The Out Of Sample statistic are not reliable as the IS statistic will be the results of OOS period as well 


Please see attached screen capture





Votes: +7

Drop files to upload

or

choose files

Max size: 5MB

Not allowed: exe, msi, application, reg, php, js, htaccess, htpasswd, gitignore

...
Wait please