<- Back to Tasks

Strategies take a very long time to load from disk at startup

Maybe this is normal, but it feels slow to me...on opening SQX the strategies load into the data banks....in my case I have 23,000 split over 3 banks...it takes about 40 minutes to load the 23,000 strategies before I can use SQX/start the custom projeect.



Its a lot, I know, but time per strategy doesn't change if I have 1,000 to load or 23,000.


Maybe its normal? I am using an SSD which is very fast on all benchmarks and throughout other apps and loading the OS etc.


Curious what others are experiencing and whether this is expected behavior or not?

Attachments
No attachments
  • Votes 0
  • Project StrategyQuant X
  • Type Bug
  • Status Fixed
  • Priority Normal

History

N
#1

nathan

24.03.2019 19:01

Task created

MF
#2

Mark Fric

25.03.2019 09:53

Status changed from New to Waiting for information

how big are your strategies? Maybe you have a lot of data (WFM results, etc.) stored there.


I just tested it with 5000 strategies, and they were loaded in less than a minute.


Could you attach a few of your strategies here? 40 minutes is a lot, we can checkif we can optimize the process somehow.

N
#3

nathan

25.03.2019 12:04

Attachment Strategy 11363.sqx added

Attachment Strategy 11365.sqx added

Attachment Strategy 11368.sqx added

Attachment Strategy 11369.sqx added

Attachment Strategy 11381.sqx added

Attachment Strategy 11404.sqx added

Attachment Strategy 114061.sqx added

Attachment Strategy 11406.sqx added

Attachment Strategy 11414.sqx added

Attachment Strategy 11423.sqx added

Average size is 76K per strategy. 14,840 in one bank / folder on disk. 1.08GB total.


If I could load 5K strategies in less than a minute that would be amazing :)


I have attached a random sample of them here.

MF
#4

Mark Fric

25.03.2019 12:47

Attachment databank.jpg added

I just tested your strategies - cloned these files to make circa 4500 copies, and loaded them into SQ. 

It loaded within 2 minutes, and uses less than 3 GB of memory. CPu was around 4% during the load.


So I'd say there must be something wrong with your setup. 


I would suspect there is a memory problem, how big memory do you use in SQ? 

If SQ reaches the upper boundary it starts alocating and deallocating the memory and it could slow things down a lot.


Or, don't you have file-based databanks configured in Configuration -> Databanks?



N
#5

nathan

25.03.2019 13:01
I have 128GB ram installed, with 110GB allocated to SQX in SQX settings.


'Use file-based databanks' is tunred off (recommedned). Is 'on' better?


I have tried this also on the Dell OEM SSD for the server, and also on a Samsung EVO 970 SSD, both give very simialr load times/results.


MF
#6

Mark Fric

25.03.2019 14:38
It should be turned off. I don't understand the slowdown, I have also Samsung 970 Evo SSD disc. It is not related to disc at all, there must be something that is causing the slowdown.


Could you send me a screenshot of a heap chart from SQ and CPu chart from Task manager during the load?


In the worst case if you want I can try to connect to your server remotely and profile it durng the load to find the problem.

N
#7

nathan

25.03.2019 17:10
Here you go


https://www.dropbox.com/s/nz4dsjfp33bmj9t/Slow%20Load.mov?dl=0


Heap Chart in SQX from 05:09 onward


MF
#8

Mark Fric

26.03.2019 10:27
I played with it for a few hours, but it still works fast for me. I put 5000 strategies each to 5 different custom projects, SQ was started and loaded within 5 minutes.


So I don't understand what could cause the slowdown in your case. Would it be possible to investigate it on your server or deskrtop remotely for example using TeamViewer?

I'd need to install Java profiler there.


N
#9

nathan

26.03.2019 16:51
Thats fine with me, Teamviewer or however you prefer.


My gut is telling me it is link with the UI freezes in this build; they last about 5 to 10 seconds, every 5 minutes or so, but as the startegy numbers in the databanks have increased the frequency of the freezes has steadily increased...they are now at about once every minute or so. Correlation is not causation of course.


PM me when you would like access and I will provide it.

MF
#10

Mark Fric

28.03.2019 14:09

Status changed from Waiting for information to Fixed

I found the problem, it was that the startegies had also their Optimization profile data saved. And with 10-20.000 optimizations per strategy this could be a lot of data.


I improved file format, resulting in 2-3x speedup in loading strategies. This is maximum that can be done for now without making bigger modifications to the system.

l
#11

Loonly

08.04.2019 20:26
To solve the issue i suggest to re-test all the strat and resave them, in such a way they will loose the WFM optimization, and the file size will be smaller
Votes: 0

Drop files to upload

or

choose files

Max size: 5MB

Not allowed: exe, msi, application, reg, php, js, htaccess, htpasswd, gitignore

...
Wait please