Please consider to parallelise the Creation of backtest data feed (it takes hours)

When we retest forex pairs on Additional Markets tick by tick, it takes hours to prepare the whole "backtest data feed".
That's happen with cpu utilization @ 3% !
Could you workaround this issue working the data feed creation backtest on all cores available?
image 28 markets (all the fx pairs)  tick by tick ...made one after one and not in parallel... really annoying process.



Thank you.

Attachments
  • Votes +6
  • Project StrategyQuant X
  • Type Feature
  • Status New
  • Priority Normal

History

l
#1

Loonly

05.02.2022 20:04

Task created

b
#2

bentra

05.02.2022 20:09
Voted for this task.
g
#3

geektrader

06.02.2022 01:43
Voted for this task.
g
#4

geektrader

06.02.2022 01:45

Attachment Untitled.png added

Attachment Untitled2.png added

Untitled.png
(508.92 KiB)
Untitled2.png
(432.49 KiB)
I absolutely back this up, this is also the case when importing tick-data in the Data Manager. Right now, I am importing tick-data for 4 pairs at once, yet it uses just 1 CPU core, and the disk is not maxed out either at 34 MB/s for writing (it´s a NVME with 3 GB/s). The import of 28 pairs with tick-data is taking 2 days now! This MUST be parallelized.
Bt
#5

beetrader

06.02.2022 10:03
Voted for this task.
g
#6

geektrader

07.02.2022 06:41

+24 hours later and the files are still importing :-(


Just to put this into perspective: exporting one of these tick-data files from MT5 took about 10 minutes each. So a total of 40 minutes to export them, and currently 24 hours+ to import them to SQX, and it´s still running.

g
#7

geektrader

07.02.2022 14:49

It´s also absolutely crazy what amount of LOG files StrategyQuantX generates during the import. Importing these 4 pairs created 130GB of log files:


07.02.2022  13:41   131.569.745.797 log_2022_02_07.log
               1 File(s) 131.569.745.797 bytes
               0 Dir(s)               0 bytes free


So that even my disk even went full while importing.


This clearly needs a fix by letting us set the max size of the log file, e.g. 64MB (as an example). After it reaches the max size, the oldest entries are replaced.

E
#8

Emmanuel

09.02.2022 14:42
Voted for this task.
b
#9

bentra

10.05.2022 17:15


it is faster to backtest on many istances of MT4  with TDS than SQx . in-fact if i want to backtest a strategy that can trade 13 markets , in mt4 it takes few minutes using 13 istances, while on SQx wich is 64 bits and multicores, it takes 34 minutes: 13 markets 2007-2022 retest tooks 34 minutes on a fast cpu on MT4 on a slow cpu using 13 istances it took 10 minutes: 


l
#10

Loonly

10.05.2022 17:31
Voted for this task.
f
#11

FirestarZA

16.05.2022 16:08
Voted for this task.

Votes: +6

Drop files to upload

or

choose files

Max size: 5MB

Not allowed: exe, msi, application, reg, php, js, htaccess, htpasswd, gitignore

...
Wait please