SQX exports MT5 tick data in the wrong format causing subtle corruption

UPDATE:
It is clear now that SQX does not export MT5 tick data in the correct format.***

Here is what SQX forex/index/commodities export to mt5 tick data looks like currently. This is the wrong way to do it which messes up the data during import:

2003.05.05 03:00:05.536,1.12161,1.12177,1.12161,14,15.0
2003.05.05 03:00:06.561,1.12166,1.12182,1.12166,1,15.0
2003.05.05 03:00:14.085,1.12163,1.12183,1.12163,1,20.0

Here is the correct way to export forex/index/commodities tick data for MT5 terminal 

2022.02.07 02:00:00.192,1.14545,1.14549
2022.02.07 02:00:00.294,1.14546,1.1455
2022.02.07 02:00:00.396,1.14547,1.1455



Notice it is simply: "date,bid, ask"
The volume of 1 is implied in forex since it is tick data, there is no "last" in forex, only bid and ask are tracked. The flags are left up to MT5 to auto calculate. I specify "forex/index/commodities" because MT5 stocks and futures might be different.

NOTE: m1 data export from SQX to MT5 seems fine, it is just the tick data that is in the wrong format.

---------------------------------
***Since you can see in the screenshots below the malformed tables during the import of the SQX tick data to MT5. Also, the TDS exported forex/indices/commodities tick data (exported to mt5 in mt5 format) is matching exactly SQX own internal data and SQX exported to MT5 data is not even matching its own internal data.  TDS is obviously exporting to MT5 tick data in the correct format. So SQX needs to export forex/indices/commodities to MT5 data in exactly the way TDS does. 
------------------------------------

This ticket started out as a "difference between MT5 and SQX backtest" ticket. However, I have discovered the source of the differences are because my tick data in MT5 was corrupted by SQX tick data exporter.


First I found out that when I ran the troubled strat on m1 sim in MT5, it matched nicely the m1 sim in SQX. Not only that but real ticks in sqx matched up fine (at least the order levels, SL and TP etc did) with m1 sim mode in MT5. So the only thing "off" was the test in MT5 using tick data, sounds like bad tick data in MT5 right? I tried to re-download fresh then re-export and re-import the dukas tick data from SQX to mt5 but I got the exact same results. Then I tried exporting Dukas tick data from TDS and imported that into MT5 and what do you know? The strategy now matches perfectlyMT5  every tick mode vs SQX every tick mode. Therefore, the only explanation is that the SQX exporter is corrupting the tick data.


So anyway, I leave attached the strategy and screenshots of using the strategy on the corrupted data exported from SQX in MT5 vs the test in SQX. I'm really pissed because I have been using this corrupted data for over a month in all my MT5 optimizations and tests.

EDIT I HAVE RULED OUT A FEW THINGS AND PROVEN FURTHER THAT DATA FROM SQX EXPORTER IS NOT WORKING WELL IN MT5:
I have now ruled out any potential network drive issue and also tried using a fresh install of MT5. I have also tried other strategies too and so far every strategy I try gets the following result:


-a test in MT5 every tick mode using MT5 dukas tick data that was exported from SQX is always a little different result from everything else including sqx mt5 engine every tick mode.

-Mt5 dukas data exported from TDS is always matching EXACTLY the results of SQX mt5 engine every tick mode which matches up nicely with m1 sim mode of MT5 which matches up nicely with SQX on m1 sim mode too. It is only when I use tick data exported from SQX that trades do not exactly match.


EDIT
I have not tested other symbols, just EURUSD.
Attachments
2022-02-11 2.png
(462.25 KiB)
2022-02-11 1.png
(105.73 KiB)
Nonq_xtraExits_Strategy 644323.sqx
(157.84 KiB)
  • Votes +8
  • Project StrategyQuant X
  • Type Bug
  • Status Fixed
  • Priority High

History

b
#1

bentra

12.02.2022 00:45

Task created

b
#2

bentra

12.02.2022 00:46
Voted for this task.
b
#3

bentra

12.02.2022 05:58

Attachment Nonq_xtraExits_Strategy 625435.sqx added

Nonq_xtraExits_Strategy 625435.sqx
(147.46 KiB)
Here's another one. Tick data results not matching but....

Very WTF: 
For this strategy, tick modeling in MT5 based on m1 matches up nicely with tick simulation based on m1 in SQX. Also, running this strat on tick modeling in MT5 based on m1 data matches up nicely with SQX real ticks real spread!!! Why is MT5 on real tick mode making the orders placed so much differently?

I tried redownloading and reexporting from SQX and reimporting to MT5. I get the same results....

I tried importing TDS data. Should be the same Dukas data but.... Now MT5 is matching SQX perfectly! So must be a bug in the data exporter of SQX!




b
#4

bentra

12.02.2022 06:14

Description changed:

This ticket started out as a "difference between MT5 and SQX backtest" ticket. However, I have discovered the source of the differences are because my tick data in MT5 was corrupted by SQX tick data exporter.


First I found out that when I ran the troubled strat on m1 sim in MT5, it matched nicely the m1 sim in SQX. Not only that but real ticks in sqx matched up fine (at least the order levels, SL and TP etc did) with m1 sim mode in MT5. I tried to re-export and re-import the dukas tick data from SQX to mt5 but I got the exact same results. Then I tried exporting Dukas tick data from TDS and imported that into MT5 and what do you know? The strategy now matches perfectlyMT5  every tick mode vs SQX every tick mode. Therefore, the only explanation is that the SQX exporter is corrupting the tick data.


So anyway, I leave attached the strategy and screenshots of using the strategy on the corrupted data exported from SQX in MT5 vs the test in SQX. I'm really pissed because I have been using this corrupted data for over a month in all my MT5 optimizations and tests.


E
#5

Emmanuel

12.02.2022 06:36
Voted for this task.
g
#6

geektrader

12.02.2022 07:17
Voted for this task.
TV
#7

Tomas Vanek

12.02.2022 12:15
Please attach some sample of data exported from SQ X
b
#8

bentra

12.02.2022 18:11

Attachment 2022-02-12.png added

2022-02-12.png
(46.94 KiB)
It's a 20g file...

I can't find any info on the 1 flag and some of the flags are uneven numbers so maybe that's something...
https://www.mql5.com/en/forum/77002


Also, in forex spot I thought the volume of one tick is always one, from tds the volume column is empty. Also, I thought the 32 and 64 and 8 and 16 flags are not tracked in forex but somehow those are sometimes flagged in the sqx export. In fact from TDS looks like the flags are only the bid and ask flags and only the bid and ask and flags columns have numbers nothing else and flags is always 6. Not sure if that's what the issue is but might be something to look into.

TV
#9

Tomas Vanek

14.02.2022 09:36
you do not have sent hole file, we need just a sample of 100 lines
b
#10

bentra

14.02.2022 16:50
How do you want me to do this? I just have windows 10 and some trading tools installed.... What's the easiest way to extract 100 lines from a 20g csv file with windows?
b
#11

bentra

14.02.2022 17:27
I have exported a 1 month csv from SQX and emailed it in to support.
b
#12

bentra

14.02.2022 19:22
I have now ruled out any potential network drive issue and also tried using a fresh install of MT5. I have also tried other strategies too and so far every strategy I try gets the following result:


-MT5 dukas tick data that was exported from SQX is always a little different result from everything else including sqx mt5 engine every tick mode.

-Mt5 dukas data exported from TDS is always matching EXACTLY the results of SQX mt5 engine every tick mode which matches up nicely with m1 sim mode of MT5 which matches up nicely with SQX on m1 sim mode too. It is only when I use tick data exported from SQX that trades do not exactly match.

b
#13

bentra

14.02.2022 19:28

Description changed:

This ticket started out as a "difference between MT5 and SQX backtest" ticket. However, I have discovered the source of the differences are because my tick data in MT5 was corrupted by SQX tick data exporter.


First I found out that when I ran the troubled strat on m1 sim in MT5, it matched nicely the m1 sim in SQX. Not only that but real ticks in sqx matched up fine (at least the order levels, SL and TP etc did) with m1 sim mode in MT5. So the only thing "off" was the test in MT5 using tick data, sounds like bad tick data in MT5 right? I tried to re-download fresh then re-export and re-import the dukas tick data from SQX to mt5 but I got the exact same results. Then I tried exporting Dukas tick data from TDS and imported that into MT5 and what do you know? The strategy now matches perfectlyMT5  every tick mode vs SQX every tick mode. Therefore, the only explanation is that the SQX exporter is corrupting the tick data.


So anyway, I leave attached the strategy and screenshots of using the strategy on the corrupted data exported from SQX in MT5 vs the test in SQX. I'm really pissed because I have been using this corrupted data for over a month in all my MT5 optimizations and tests.

EDIT I HAVE RULED OUT A FEW THINGS AND PROVEN FURTHER THAT DATA FROM SQX EXPORTER IS NOT WORKING WELL IN MT5:
I have now ruled out any potential network drive issue and also tried using a fresh install of MT5. I have also tried other strategies too and so far every strategy I try gets the following result:


-a test in MT5 every tick mode using MT5 dukas tick data that was exported from SQX is always a little different result from everything else including sqx mt5 engine every tick mode.

-Mt5 dukas data exported from TDS is always matching EXACTLY the results of SQX mt5 engine every tick mode which matches up nicely with m1 sim mode of MT5 which matches up nicely with SQX on m1 sim mode too. It is only when I use tick data exported from SQX that trades do not exactly match.


EDIT
I have not tested other symbols, just EURUSD.

HH
#14

Hans

15.02.2022 00:05
Voted for this task.
TH
#15

Tomas Hruby

16.02.2022 09:35
bentra: "I have exported a 1 month csv from SQX and emailed it in to support." It is under ticket SZS-MPGQP-545
b
#16

bentra

07.03.2022 23:48

Attachment 2022-02-12.png added

2022-02-12.png
(46.94 KiB)
You can even see in this screenshot that MT5 doesn't like the SQX export formatting because there is a row near the bottom of this screenshot that contains only a volume with no price data lol.
CG
#17

Chris G

21.03.2022 17:54
Voted for this task.
k
#18

Karish

24.06.2022 13:50
Voted for this task.
IN
#19

altenburg90

04.07.2022 12:30
Voted for this task.
KB
#20

kbtech

07.08.2022 04:56
Voted for this task.
b
#21

bentra

01.09.2022 17:10
I think that there are multiple mt5 data formats, I think SQX currently exports correctly for stocks but I'm certain it is currently incorrect for forex and cfds (indexs, coms) of MT5.
TT
#22

Tamas

13.09.2022 09:16

Priority changed from Normal to High

PS
#23

Petr Šomek

14.09.2022 08:59

Status changed from New to Fixed


Votes: +8

Drop files to upload

or

choose files

Max size: 5MB

Not allowed: exe, msi, application, reg, php, js, htaccess, htpasswd, gitignore

...
Wait please