MC modified randomize history data the data is not actually being fed in to the indicators and price levels calculations!

When using the data randomizer for some reason market order strats behave as expected but stop order entry orders are very very messed up. With a very tiny change per tick set on "only this timeframe precision" I have screenshotted a typical market order strat and a typical limit order strat. There should be only a tiny change to OHLC on the main time frame.

As you can see the limit order strat data gets absolutely mangled and the market order strat seems to behave closer to what's expected with only small changes to the data. (Using larger changes messes limit/stop order strats mc tests up even more beyond recognition.) This is actually happening on higher precisions as well. This makes me think SQX may be drawing the non modified (by the data randomizer) data for the indicators (cache issue?) and using the randomized data for only the resulting orders' execution. EDIT: Yes this is definitely what's happening, the indicators are not using the randomized data properly, proof bellow.





Attachments
typical_limit_stop_strat.png
(994.71 KiB)
typical_market_strat.png
(865.30 KiB)
  • Votes +11
  • Project StrategyQuant X
  • Type Bug
  • Status Fixed
  • Priority High

History

b
#1

bentra

18.04.2021 00:38

Task created

b
#2

bentra

18.04.2021 02:53

Subject changed from MC data randomizer market orders vs stop/limit orders huge difference to MC data randomizer market orders vs stop/limit orders = huge difference

Description changed:

When using the data randomizer for some reason market order strats behave as expected but stop order entry orders are very very messed up. With a very tiny change per tick set on "only this timeframe precision" I have screenshotted a typical market order strat and a typical limit order strat. There should be only a tiny change to OHLC on the main time frame.

As you can see the limit order strat data gets absolutely mangled and the market order strat seems to behave as expected with only small changes to the data. (Using larger changes messes limit/stop order strats mc tests up even more beyond recognition.) This is actually happening on higher precisions as well. This makes me think SQX may be drawing the non modified (by the data randomizer) data for the indicators (cache issue?) and using the randomized data for only the resulting orders' execution. Either that or the tic randomizer functions are just getting called way more frequently for the stop order strats than they do for the market order strats for some reason. Please look in to it.

EDIT: I just confirmed the data randomizer snippet gets called the same amount of times no matter if it's a market order strategy or a limit order strategy. Also noticed the higher the % change the fewer orders are executed on the limit/stop strats which is not right. Feels like the stop/limit orders are getting further and further away and don't get triggered anymore like the indicators aren't matching up with the randomized data.




b
#3

bentra

18.04.2021 02:56

Subject changed from MC data randomizer market orders vs stop/limit orders = huge difference to MC modified randomize history data with market orders vs with stop/limit orders = huge difference

b
#4

bentra

18.04.2021 03:03
Voted for this task.
b
#5

bentra

18.04.2021 04:02

Attachment Screenshot 253.png added

Attachment Screenshot 252.png added

Attachment buggedmcStrategy 1217978.sqx added

Attachment buggedmctestbuggyrandomizer.sqx added

buggedmcStrategy 1217978.sqx
(56.28 KiB)
Screenshot 252.png
(649.94 KiB)
buggedmctestbuggyrandomizer.sqx
(57.14 KiB)
Screenshot 253.png
(556.05 KiB)
I have proven that the resulting randomized data is not actually being fed in to the indicators calculations properly.

I will attached two separate market entry strategies which are nearly identical except one has a very nearly the same SL except the SL is tied to an ema (1,1) +/- atr (for bug finding) instead of just a regular ATR stop. The other has the ATR based stop. Both do similar equity curve in the main data. The one with the SL tied to an indicator is clearly showing signs in the MC test that the SL distance is getting absolutely mangled. There is almost nothing different about these two strategies, there should be almost no difference during the MC test which randomizes data. The introduction of a SL based on a price level indicator messes it right up!
b
#6

bentra

18.04.2021 07:44

Attachment Screenshot 254.png added

Screenshot 254.png
(317.80 KiB)
It's easier to see telling patterns in the equity curves when we make it fixed lot size. Here we see again the EMA (1,1) +/- 3 x ATR SL strategy. In some of the MC data randomization iterations you can see strings of tiny losses and also large gaping single trade losses.  This SL is not staying 3 or 4 ATR away like it is in the main data like it should be.

In the screenshot in my previous post you can see the opposite because it is risk %. Some huge wins because of tiny SL which causes large lot size and some tiny wins because of huge SL which caused small lot size.


IH
#7

clonex / Ivan Hudec

18.04.2021 08:15
Voted for this task.
b
#8

bentra

18.04.2021 08:38

Attachment Screenshot 255.png added

Attachment longonlytestbuggyrandomizer.str added

longonlytestbuggyrandomizer.str
(82.32 KiB)
Screenshot 255.png
(555.63 KiB)
One more for good measure. I changed the  EMA (1,1) +/- 3 x ATR SL strategy to be long only instead of symmetrical and so a single iteration either has a SL way too far away with large losses or way too close with long strings of tiny losses, it's even easier to see! (As before, the SL should be staying about 2 to 4 ATR away)
o
#9

Enric

18.04.2021 09:21
Voted for this task.
JH
#10

Jabezz

18.04.2021 09:51
Voted for this task.
k
#11

Karish

18.04.2021 11:45
Voted for this task.
b
#12

bentra

18.04.2021 19:16

Attachment longonlytestbuggyrandomizerclose.sqx added

Attachment Screenshot 256.png added

Screenshot 256.png
(659.94 KiB)
longonlytestbuggyrandomizerclose.sqx
(52.24 KiB)
Even when the SL is based on close[1] +/- 3 atr instead of ema [1,1] +/- 3 atr.... same thing.

Even basic price levels are using the original data instead of the randomized price data!
h
#13

hankeys

19.04.2021 07:42
Voted for this task.
b
#14

bentra

20.04.2021 00:52

Attachment buggfinder-logging.sqx added

Attachment SQExtension-closeclone.sxp added

buggfinder-logging.sqx
(52.13 KiB)
SQExtension-closeclone.sxp
(1.32 KiB)
My thought process is a little hard to follow but I make it easy, with this custom strat and this closecloned block with logging you can see in the logs any given bar# is always the same close price in any MC iteration, it's the same as the main data! 8311 is a bar number I picked randomly and 0.89465 is the bars close price it is not being affected by the data randomizer.

...
...
18:18:26.321 [Blocking computeThread common #0 - longonlytestbuggyrandomizer0] INFO  SQ.Blocks.Price.CloseClone - closeclone: 8311 0.89465
...
...
18:18:31.166 [Blocking computeThread additional #0 - longonlytestbuggyrandomizer MC 5] INFO  SQ.Blocks.Price.CloseClone - closeclone: 8311 0.89465
...
...
18:18:31.170 [Blocking computeThread additional #4 - longonlytestbuggyrandomizer MC 9] INFO  SQ.Blocks.Price.CloseClone - closeclone: 8311 0.89465
...
etc.



JK
#15

Insanity82007

20.04.2021 02:53
Voted for this task.
JK
#16

Insanity82007

20.04.2021 02:59
This is awesome Bentra. I've always thought the MC Randomize History methods in SQ aren't properly implemented. One of the things that bothers me the most is how closely each equity curve follows the shape of the main equity curve, even in the messed up equity curves in your pictures above. This would also support the idea that it's not retesting on the indicators, but modifying orders instead.


If it was truly randomizing data correctly and running true backtest, the results would more closely resemble what's happening with backtests on additional markets because the price data is randomized. 


I also believe that if we have the randomize price history monte carlo testing CORRECTLY implemented in SQ, then we won't ever need to use the backtests on additional markets feature because we can test HUNDREDS or THOUSANDS of randomly generated price datasets at a time instead of having to define EACH and EVERY other market we want to use

e
#17

eastpeace

20.04.2021 04:19
Voted for this task.
MF
#18

Marti

20.04.2021 06:17
Voted for this task.
b
#19

beppil

20.04.2021 08:57
Voted for this task.
m
#20

Martin

22.04.2021 23:18
Voted for this task.
MF
#21

Mark Fric

12.05.2021 13:07

Status changed from New to In progress

Priority changed from Normal to High

I confirm this behavior, I'm looking at it.
MF
#22

Mark Fric

19.05.2021 11:35

Status changed from In progress to Fixed

I finally found the problem, it was a bad setting. The changes in history data were used in SL/PT but they were not reflected in data that was sent to indicators

Votes: +11

Drop files to upload

or

choose files

Max size: 5MB

Not allowed: exe, msi, application, reg, php, js, htaccess, htpasswd, gitignore

...
Wait please