109 - SQ runs slow after many hrs, even after exit and start again.

SQ4 Build 109 - After more than 10 hours of continuous strategy building, SQ4 slows the whole system down considerably. After stopping the build, clearing memory (which was less than 50% of the allowed 120 GB), and restarting the build had no effect. Exiting SQ4 and re-starting it had a minimal improvement.  A windows re-boot was required to restore the program's operation back to normal.

My system:
Dual Xeon CPUs, 16 cores, 32 threads, 128GB ram SQ4 running on a SSD
Only a CPU temperature & utilization monitor running in the background.

SQ4 Performance settings:
1 core dedicated to UI
120GB (of 128GB) allocated to SQ

Symptoms:
  • The mouse does not appear at first then minutes afterwards, functionality returns very slowly but never recovers completely.
  • The generation of strategies slowed from over 40,000 per hour to about 25,000 per hour.
  • Clicking on anything in SQ generates a warning in the top left corner that SQ is not responding. It eventually responds after time.

The only true cure is to re-boot the computer.
Attachments
No attachments
  • Votes 0
  • Project StrategyQuant X
  • Type Bug
  • Status Refused
  • Priority Normal

History

d
#1

Dave

04.09.2018 01:53

Task created

MF
#2

Mark Fric

04.09.2018 09:06

Status changed from New to Waiting for information

this looks more like problem on OS level. 


Do you really need 120 GB of RAM for SQ?

How many strategies do you store into databank?


Using this extreme amount of RAM doesn't necessarily bring better performance. I would recommend using jst 16 - 32 GB of memory, this should be enough for mos scenarios.

Memory management will be slow when so much memory has to be managed, and it seems it will bring problems on OS level.

 

d
#3

Dave

04.09.2018 19:18

Attachment SQ4 Memory usage.jpg added

This Lenovo Workstation was custom built to run 28 instances of SQ3 each limited to 4GB ram and was set running for 2 - 4 months at a time without any intervention or supervision. It is a solid and stable data center quality computer and even though it uses old tech and just won't die.


With SQ4's advanced testing and better usage of resources I agree that 128 GB is overkill but it is paid for and I have no other use for the memory. I use a faster newer tech single CPU machine for analyzing the results. 


I have the databank set to store up to 10,000 strategies but I have not produced more than 20 at this time - thanks to the advanced checking & crosschecking in SQ4.


After the last re-boot it has now been running over 17 hours and the computer is still responsive as it should be.  SQ4 reports memory heap as:

Total: 120GB

Green shaded area is at about 58GB

Orange (memory in use): up to about 56GB


While this computer is the most stable that I have, it is always possible that the problem was a random event with the OS.


Dave

MF
#4

Mark Fric

06.09.2018 08:27

Status changed from Waiting for information to Refused

you have quite an extreme configuration :-)


I would still try to run it with much less memory, for example 32GB. You might find out that SQ X is actually faster in that configuration.


And you could try to run 2 or 3 instances of SQ X, maybe the combined performance would be also better.


Votes: 0

Drop files to upload

or

choose files

Max size: 5MB

Not allowed: exe, msi, application, reg, php, js, htaccess, htpasswd, gitignore

...
Wait please