SQX to Tradestation backtesting results HUGE mismatch issues B124

Since I am personally very interested in futures trading I was happy to see you have 'released' futures historical data service together with an ebook and some example strategies.

However, in the last few month I was struggling in getting SQX results synchronized with TS backtest results.

Now having your example futures strategies I performed this SQX to TS cross check again to see if my setup is correct. 

See attached figure.

Your own strategies showing some small (but acceptable) mismatch between TS and SQX results. This small mismatch is acceptable. But when I build new strategies using similar indicators and very same settings 80% of strategies build by SQX show very different results in TS backtesting.

B.T.W: I already did everything as described by:

Moreover, retesting of your example strategies with my SQX settings gives similar results as presented by your reports. This more or less proves settings are correct...

So what is going on?

See attachments for SQX settings and example of new strategy with large SQX to TS mismatch.



Attachments
ts vs sqx synch backtest.png
(517.55 KiB)
Strategy 28163.sqx
(51.98 KiB)
  • Votes +3
  • Project StrategyQuant X
  • Type Bug
  • Status Fixed
  • Priority High

History

Kc
#1

coensio

08.11.2019 22:07

Task created

Kc
#2

coensio

08.11.2019 22:07
Voted for this task.
Kc
#3

coensio

10.11.2019 10:43

Attachment Strategy 25127.sqx added

Attachment Strategy 35163.sqx added

Attachment SQX to TS issues B124.png added

Added another few examples of backtest issues between SQX and TS.

There must be something seriously wrong with the (my) settings or Tradestation engine inside SQX.


Please verify.

MF
#4

Mark Fric

11.11.2019 07:43

Priority changed from Normal to High

a
#5

aass

11.11.2019 12:28
Voted for this task.
MF
#6

Mark Fric

11.11.2019 15:31

Status changed from New to Waiting for information

Krzystof, when I retest your strategies I get almost identical results.


Few points:


- you tested it with B124, there were some bugs that will be fixed in B125


- slippage settings for Tradestation engine in SQ will also change, it will be configured in $, just like in Tradestation.

But if you see differences try to retest it first without any commisisons and slippage in both platforms.


- from your config I don't know what @ES data exactly you used in SQ.
Did you use our SQ futures data for @ES in SQ, or did you export data from your Tradestation and import them to SQ?


- I see you use NoSession in SQ, this would be ok for data imported from Tradestation , but not if you use SQ data. 

You cannot set No session in Tradestation as far as I know. Which session exactly did you use in Tradestation?



Kc
#7

coensio

11.11.2019 17:01

Indeed I'm using TS exported data (exchange time). So I use 'no session' option since sessions are already integrated in the chart data.


So today I built again, using builder with 0 commissions and no spread (=0), I also tested on TS version 9.5 and 10.


So far still huge mismatch, I can send you my builder settings...you can build and see that the SQX results do no match.


P.S: I do not even use Inside bar testing option in TS.




MF
#8

Mark Fric

12.11.2019 08:21
please send me screenshot of your settings in TS - symbol settings screen and session.


I'll be able to reproduce it with that.


You could also attach few more strategies that are different.

Kc
#9

coensio

12.11.2019 10:48

Attachment Futures_intraweek_config TS errors.cfx added

Attachment Strategy 37175 TS compile errors.sqx added

Attachment Strategy 38152 TS compile errors.sqx added

#ISSUE 1:

Ok I think I found the 'root-cause' behind what was going on.....in this case it seems that the issue lies on the TS side. You can easily reproduce it by:

1. Opening a new strategy e.g.: 'SQXtest' in TS Dev. Env. window.

2. Copying and pasting a SQX generated code from 'Source code' tab

3. Performing the first backtest (it will be always OK)

4. Then (this was the issue) copying a new strategy from another SQX generated system also using the 'Source code' tab and pasting it to the same 'SQXtest' in TS Dev.Env window and so overwriting the first strategy code. Then recompiling new code and watching the TS report window refreshing...

5. The results from the 2nd strategy will be always corrupted.


Solution:

Always make a new fresh strategy file/name in TS Dev. Env. window for each new SQX generated code.


#ISSUE 2:

Moreover, I have 0 proof for that since it is very hard to reproduce, but I get the feeling that sometimes after a very long SQX operation the commission settings / slippage settings are not getting updated when manually changed between different strategy building runs...but again I have 0 proof for this issue, but maybe you already have seen something similar...


#ISSUE3:

Some of the strategies produced by SQX do not compile on TS, this time I am attaching a example builder which will produce strategies with TS compile errors (together with two example strategies). When compile errors are manually fixed, the results are totally different between SQX and TS.










Kc
#10

coensio

12.11.2019 12:33

Attachment Strategy 111146 TS compile errors.sqx added

Attachment TS code compiler issues B124.png added

Attached another example of compiling + TS vs SQX backtest synchronization issues.


Kc
#11

coensio

12.11.2019 12:58

Attachment TS sync issue 2.png added

Attachment Strategy 36186 TS sync error 2.sqx added

Attached another example of a strategy without TS compiling errors ('out-of-the-box') and yet with a totally different backtest results. Same settings as in previous example used, nothing changed.


You you need more info just let me know...



e
#12

eastpeace

12.11.2019 13:21

I suggest that SQ show the commissions and slippage cost in strategy performance overview.


The cost is calculated by roundturn, and TS/MC withdraw that per side. That is the key point. Pips or in $ ,that is OK.


Some strategy I test in MC without commissions and slippage, the net profit is little better than that in SQ. ( no commissions and slippage in SQ too.)


And another issue may be the reference bars, the indicator periods in SQ. the default value is 200.  The first few trades may be different, depending on the maximum period actually used in the strategy. 

Kc
#13

coensio

12.11.2019 14:12

Attachment Strategy 36182 TS sync No error .sqx added

Attachment TS sync no issue.png added

Finally attached strategy 3.6.182 which is OK and is very, very similar to 3.6.186 which was NOK, (see previous post).


If you look in the code: Str 3.6.186 handles 'LongOpenShift' in different way than 3.6.182. Maybe this mechanism or some related TS configuration setting has something to do with this issue.

e
#14

eastpeace

15.11.2019 14:16
Voted for this task.
Kc
#15

coensio

19.11.2019 12:30

Just FYI: issue can also be reproduced in SQX B125.


P.S.: I also did not manage to synchronize MT5 results with SQX results using exported futures data.




MF
#16

Mark Fric

19.11.2019 13:35
I fixed the problem with compilation errors.


As for problem that only first backtest is always right - it is TS issue, it rememebers the parameters of a strategy.


So let's say you create new strategy StrTest wiht parameter Param1=10. When you'll add it to chart, it will be computed correctly.


Then you get a source code of another strategy from SQ and copy & paste it to StrTest in TS editor and click on Verify. The strategy is compiled.

This strategy has uses same Param1, but with value 20.


When you'll look at chart, the second strategy was applied, but Tradestation kept teh same parameter values from the previous strategy, so Param1 will be still 10.


There are two different solutions;

1. create new strategy for every retest


2. modify the strategy code and change

inputs:

to 

vars:


in the beginning of the strategy. This will make all parameters to be unconfigurable variables, and TS will have nothing to remember.

Kc
#17

coensio

19.11.2019 15:31

Ok great to see some progress. Yes indeed I've already figured out that the part of the issue was caused by TS compiler and variables.


Did you succeed with reproduction of the second part of the issue? SQX to TS differences even with strategies that do compile well? I see also similar issues with MT5 backtests.

 

MF
#18

Mark Fric

27.11.2019 15:00
I resolved all the issues specifically mentioned in this task. I compared your strategies, got the same results between SQ and TS.


If you have more strategies where trading is very different please attach them here.


Also, please open separate task for MT5 problems.

Kc
#19

coensio

27.11.2019 17:35

Attachment Strategy 49108.sqx added

Attachment SQX to TS sync issue 3.png added

Please see and verify the attached strategy example. @YMD.D M15 symbol.
MF
#20

Mark Fric

28.11.2019 09:28

Attachment ymd_mine.jpg added

I just tested it in my Tradestation with @YM.D M15 data, Regular session 8:30 Am - 3:15 PM, $5 commission per contract.


Equity curve in TS seems to be as same as in SQ.


So your differences are caused by something else - you must have some differences in test between SQ and TS, or changed strategy parameters.

Kc
#21

coensio

28.11.2019 09:49

Attachment TS settings YMD.png added

Can you check if the TS setting are the same? Please see the attachment.
Kc
#22

coensio

28.11.2019 09:53

Attachment zYMDM15 Strategy 11124.txt added

And to be sure about the strategy settings I'm also attaching strategy EL code.
MF
#23

Mark Fric

28.11.2019 11:08

the source code you sent is from totally different strategy, not from Strategy 49108.sqx that you attached before.


Check the entry conditions, they are different.

Kc
#24

coensio

28.11.2019 11:45

Attachment Strategy 49108 SQX to TS.png added

Attachment Strategy 49108.txt added

Ohh my mistake. Attached the proper source code file.


I'm pretty sure it's due to some TS settings..when changed I got almost exactly the same results as SQX but still no 1-on-1 match and still different than your backtest. See also attached figure.


The question is then: what TS settings should we use to synchronize it with SQX, can you share setting you have used?


Thanks.

Kc
#25

coensio

28.11.2019 15:37

Attachment Strategy 49108 SQX to TS EOD close issue.png added

Ok I found another issue. After some investigation I found out that sometimes the TS strategy does not respects the EOD exit that is specified by SQX e.g.: at 15:00h.

See example screenshots...you can easily reproduce it using strategy 4.9.108.


Is there any Ts setting that needs to be set to force EOD close? Or is this due to some source code errors on SQX side?


Kc
#26

coensio

28.11.2019 17:05

Please read this:


"The TradeStation Platform does not trigger the End-Of-Day exit on shortened trading days (e.g. the day before ThanksGiving). Users need to identify in advance when a shortened trading day will take place (usually before a Holiday or long weekend) and then on that day, the user will have to manually exit the strategy a few minutes before the market closes."


I think the best practice would be if SQX would behave EXACTLY the same as TS.


Can you fix this?


Thanks


e
#27

eastpeace

09.12.2019 15:31

Attachment Strategy 432145.sqx added

Attachment DCEi HOT 1 .7z added

Attachment instruments.png added

Attachment session in mc.png added

Attachment session.png added

Attachment lower in Mc too.sqx added

Attachment lower in MC.sqx added

Attachment Mc profit higher than SQs.sqx added

Here is a test with b126 and Multiharts. (str.. 432145.sqx)


In Multiharts

-----------------------

2%% commision
50 slippage
max reference bars 100
 net profit 63623.16
trades 59
wins% 45.76%

-------------------

no cost
net profit 69700.00
trades 59
wins% 47.46%


--------------------------

SQ
commision 4%%
slippage: 100
net profit 76276.1$
trades 63
wins% 53.97%

------------------------

no commisions and no slippage
net profit 90250
trades 63
wins% 58.06%


I have imported the latest pla file from SQ b126.

I have checked the price data in SQ and MC are same except volume.


Reasons I guess:

1, Some indicator caculattion.


Few strategy in MC have more profit than in SQ. But most are lower. (with no cost)


2, Cost draw by per side or roundturn. when the commissions is withdrawed by % of contract value type especially.

MF
#28

Mark Fric

15.05.2020 14:58

Status changed from Waiting for information to New

TB
#29

Tomas Brynda

18.05.2020 09:01

Status changed from New to Waiting for information

Hi Eastpeace,

I think that maybe the problem is in the session you use. 

It is complicated and I can't even set up the session in MC, it tells me either "Set up session end" or "Trading session can't be longer than 24 hours".

Please try to make a backtest with some regular session.


Best regards,

Tomas

e
#30

eastpeace

19.05.2020 12:45

Attachment 282154.png added

Attachment 452161.png added

Attachment Retest config.cfx added

Attachment Strategy 282154.sqx added

Attachment Strategy 38119.sqx added

Attachment Strategy 452161.sqx added

Attachment ta_hot_adjust.csv added

Very strange, the performance of most strategies in SQ and MC is not the same.


100% same price daily data.


fees in MC, is 6$ per per side.



strategy generated by B128


TB
#31

Tomas Brynda

21.05.2020 08:38

Status changed from Waiting for information to Fixed

There was a problem in history data handling in SQ.

It is fixed and will be working fine in a next release.


Thanks a lot guys for your reports.


Tomas


Votes: +3

Drop files to upload

or

choose files

Max size: 5MB

Not allowed: exe, msi, application, reg, php, js, htaccess, htpasswd, gitignore

...
Wait please