SPP Retest inconsistency

Upon doing retest on SPP, I have observed the inconsistency of results which does not happen from previous build.



I have a folder of generated strategies loaded to sqx in 2 different databanks. Then did SPP retest on both databanks (which basically have the same list of strategies). But the retest gave inconsistent results everytime. Attached is a screenshot of different results shown I did with SPP retest. (Results to BTC SPP 2 ; BTC to BTC SPP)






Attachments
image.png
(65.86 KiB)
  • Votes +11
  • Project StrategyQuant X
  • Type Bug
  • Status Refused
  • Priority Normal

History

SA
#1

shin38

14.04.2021 03:21

Task created

k
#2

Karish

14.04.2021 11:06
Voted for this task.
m
#3

Matze

14.04.2021 19:14
Voted for this task.
MF
#4

Marti

20.04.2021 06:05
Voted for this task.
NS
#5

nicksim80

21.04.2021 03:26
Voted for this task.
j
#6

ScubaJosh

21.04.2021 04:38
Voted for this task.
AA
#7

AngelDuz

21.04.2021 04:39
Voted for this task.
AC
#8

A Casey

21.04.2021 06:04
Voted for this task.
DC
#9

anuffi

21.04.2021 07:02
Voted for this task.
JH
#10

jakehobbs

21.04.2021 12:37
Voted for this task.
CG
#11

Chris G

21.04.2021 21:04
Voted for this task.
MF
#12

Mark Fric

04.05.2021 09:45

Status changed from New to Waiting for information

can you attach your custom project?


I don't see what setting and filtering you use.


I don't think this is a bug - depending on your configuration the optimization is probably genetic, and the exact result it is dependent on randomly generated first generation. So every optimization run could be different.


It is possible that you optimize by too many parameters (have too many total combinations) or have too little Max optimizations steps configured - in both cases it searches very big parameter space with limited total number of optimizations.



SA
#13

shin38

04.05.2021 10:28

Attachment 3.png added

3.png
(28.68 KiB)
Attached was the SPP setting I did. I believe it is a bug since I compared it to B130. I have been switching on both builds to see the difference. Yes SPP retest may give different set of result but I think it shouldn't give a very big difference. Also, I ran the retest as a task not a whole custom project.  (databanks SPP and BTC on the screenshot have the same data/list of strategies)
MF
#14

Mark Fric

04.05.2021 10:32
can you please also attach also screenshot of your SPP Settings screen? or the whole Retester config?
SA
#15

shin38

04.05.2021 10:47

Attachment 3.png added

3.png
(57.47 KiB)
MF
#16

Mark Fric

04.05.2021 11:02
I mean the tab Settings - I need to see how your optimization is configured. It will be really simpler if you could attach at least one strategy or Retester config. I'm not sure I'll be able to reproduce the problem, and if I'll not reproduce it I'll be not able to fix it.
SA
#17

shin38

04.05.2021 11:21

Attachment Strategy 11127.sqx added

Strategy 11127.sqx
(348.42 KiB)
MF
#18

Mark Fric

05.05.2021 12:32
I still cannot reproduce this, I believe it is a difference in configuration between your two retest tasks.


In my tests the number of strategies that pass SPP is different, and it is because of the reasons I explained - too little maximum optimization steps compared to number of possible configuration.


Unfortunately your strategy didn't contain the SPP setting you used, so I still don't know the exact configuration of your retests.


Can you attach the configs of your Retest tasks?

SA
#19

shin38

05.05.2021 13:29
There is no difference in retest configuration I am sure of. Because I just repeated the SPP retest to see how inconsistent the results will be. Again, I want to point out that I did the SPP retest task on 2 separate databanks but those 2 databanks have the same list of strategies.
MF
#20

Mark Fric

05.05.2021 13:35
ok, I understand, but can you attach the config of the retest task?
SA
#21

shin38

05.05.2021 14:44

Attachment SHEENA.cfx added

SHEENA.cfx
(130.56 KiB)
MF
#22

Mark Fric

11.05.2021 12:01

Status changed from Waiting for information to Refused

ok, I reproduced this behavior. 


When I use your config repeatedly I small differences every pass - sometimes 22, sometimes 24 strategies out of 120 pass.


But this is normal, there is no bug.

The max number of optimization steps is limited to 1000, so it doesn't do brute force optimization with all the possible outcomes, but only a selection based on genetics. It will be slightly different for every run.


If you'll increase max optimization steps to 2000 or 5000 you should see smaller and smaller variation in the results.


In my opinion 5000 optimizations is enough to make it reliable.

SA
#23

shin38

11.05.2021 12:37
"When I use your config repeatedly I small differences every pass - sometimes 22, sometimes 24 strategies out of 120 pass."


Right now, I am running on B130. And that is what I get, just a small margin of difference. But not when I used B131 before, a big number gap is the difference. I wont be able to reproduce the error now because I am still running on B130. But I hope you reconsider this error since other users have also shared that they do have problems with SPP retest on B131.


Votes: +11

Drop files to upload

or

choose files

Max size: 5MB

Not allowed: exe, msi, application, reg, php, js, htaccess, htpasswd, gitignore

...
Wait please