Custom Project : More strategies in databank make Strategy Quant Freeze every 2 minute ?

As StrategyQuant have more strategies in the databanks, it is more busy every 2 minutes even without working during idle time :

 

(capture attached)

 

In the capture 14000 strategies (in databanks), you will see a spike of CPU usage every 2 minutes by StrategyQuant, during idle time.

 

 

In the capture 35000 strategies (in databanks), 97000 strategies (in databanks), then 120000 strategies in databanks, you will see this spike becoming larger and larger.

 

This happen every 2 minutes, even if there is no running task in StrategyQuant.

 

During this spike, the UI freeze, and it makes difficult to use the UI.

 

When StrategyQuant is running, or saving , or  loading strategies, StrategyQuant stop its task to execute this unknow process.

 

When StrategyQuant have more than 50000 strategies in the databanks, in a custom projects, this take a lot more time and CPU usage.

 

 

It seem to be related to the number of strategies.

 

Is it possible to suspend this process ? or delay it ? or make it optional ?

 

It would allow us to have more strategies in the databanks, and to have StrategyQuant run faster and smoothly

 

Attachments
Capture 14000 strategies.jpg
(31.50 KiB)
Capture 35000 strategies.jpg
(29.63 KiB)
Capture 97000 strategies.png
(15.27 KiB)
Capture 120 000 strategies 3.png
(19.19 KiB)
Capture 200 000 strategies.png
(17.08 KiB)
  • Votes +3
  • Project StrategyQuant X
  • Type Bug
  • Status Fixed
  • Priority Normal

History

E
#1

Emmanuel

09.07.2020 02:09

Task created

E
#2

Emmanuel

09.07.2020 02:09
Voted for this task.
JK
#3

Insanity82007

09.07.2020 21:54
Voted for this task.
JK
#4

Insanity82007

09.07.2020 22:07
This has been an issue for a while now. It also depends on the size of the strategy files too. If you have small files e.g. builder task files with no robustness checks, it starts getting pretty shaky around 80k to 100k strategies across all Databanks in all projects.


If you have larger strategy files with robustness tests included, this can be as low as 100 strategies across databanks before it starts failing.


I think this is because SQ stores all strategies in all databanks in memory when it loads. Im not 100% sure on this though. 


I can understand the need for it to load all strategies in a single databank into memory (the one currently being worked on) as it needs to perform fitness calculations across all of these strategies while running tasks, but really strategies should only be referenced from disk when analysing then manually and should only be loaded into memory when performing tasks. Conversely they should be flushed from memory and written to disk when another task works on a different databank.

There is also the consideration that SQ may use 2 databanks in a task (source and destination), but once the need for calling strategies form the source databank is complete these should be flushed from memory and stored on disk. 


We also have MUCH faster disks these days (SSD and M.2) and SQ should evolve to make use of this and not kill the UI because all strategies in all databanks are in memory. 

JK
#5

Insanity82007

09.07.2020 22:13
To add to this... If NO tasks are running, all strategies should be referenced from disk ONLY. 

When a user double clicks on a strategy to look at it, only this strategy and contained information is loaded into memory. 

When a user double clicks on a different strategy the other strategy is flushed from memory and the new strategy is loaded into memory with contained information in memory also. 

When a task runs, the first step is to load all relevant strategies and relevant information into memory and then run the task. 


This should all be governed by an option in config settings too with 2 options:

1. Use disk and memory

2. Use memory only


This way people with more RAM can use option 2 and people with less RAM can use option 1.

People with mid range RAM and fast SSDs can also benefit a lot from option 1.


E
#6

Emmanuel

11.07.2020 14:03

I still have this issue with Build 129 rev 6


We can understand with this video :


https://drive.google.com/file/d/1Tb0HoOE4MV7LLh4vIwwODGcvdkw-8N7z/view?usp=sharing

E
#7

Emmanuel

11.07.2020 14:19
 

Thank you for your answer  Insanity82007.


  Insanity82007,  I agree , around 70-100 k stratégies, I have the same difficulties. It make it very difficult to work with SQ above 100k stratégies even with a good computer.


During the freezing period, SQ can not work properly, and we can not do anything with SQ.


 I don't see why this process is running over and over again every 2 minutes  without ever stopping doing probably doing the same loop.


Even if you are working with another custom project with empty databank, you still have freezing period from the other custom project which is not running. 


This process make SQ lose a lot of time and very dependant on the number of stratégies.


Without this freezing process, SQ could handle much more stratégies and run much faster.


Maybe, it is possible to suspend this process  or make it optional.


It would be great if we could stop/suspend this process.







E
#8

Emmanuel

16.07.2020 16:06
Thank you Mark
o
#9

Enric

21.07.2020 08:45
Voted for this task.
MF
#10

Mark Fric

22.07.2020 08:52

Status changed from New to Fixed

I fixed the CPU peak, but using SQ with tens of thousands of strategies is extreme.


All strategies in databank have to be in memory, to keep them only on disc would be very slow, even with SSD disc, and this kind of handling would bring many bugs to the process..


Votes: +3

Drop files to upload

or

choose files

Max size: 5MB

Not allowed: exe, msi, application, reg, php, js, htaccess, htpasswd, gitignore

...
Wait please