<- Back to Tasks

About Memory Configuration and sqx crashes

I conducted a test. My machine has 32g * 2 of memory. If "let program determine maximum memory" is selected in memory configuration, an error "out of memory" will be reported when the memory reaches 30g, which means that only one of the two memories can be used. After setting "set maximum memory to 60GB", two pieces of memory can be used. However, after I continue to increase the use of memory, I find that the use of memory is not limited by 60GB. After exceeding 60g, until all the memory is full, sqx software crashes. I use SQX build 131.612.
Attachments
Out of memory error.jpg
Out of memory error.jpg
(12.72 KiB)
  • Votes 0
  • Project StrategyQuant X
  • Type Bug
  • Status Refused
  • Priority Normal

History

4
#1

Jordan

25.03.2021 09:23

Task created

b
#2

bentra

25.03.2021 15:55
I doubt there is a 2x "sticks" of ram issue.
60GB you reserved for on heap but you need to have room for off heap plus everything else:
https://roadmap.strategyquant.com/tasks/sq4_7772
try 50 or 55

Also, if you're workflow/banks use more ram than you actually have there will be no settings to prevent ram problems. Holding strats in the databank can add up, try saving them to disk then empty the databanks or run smaller batches of strats through your workflow or adding more ram.
MF
#3

Mark Fric

26.03.2021 11:33

Status changed from New to Waiting for information

this has nothing to do with two RAM sticks. 


When you use "let program determine maximum memory" then Java determines the amount of max. memory it will use, you can see it as the max in the memory heap chart.


This is Java memory, SQ uses also offheap memory (outside of Java scope) for fast access, but it shouldn't take mre than 1-2 GB of it iunder normal circumstances.


YOur problem seems to be more related to your project settings, it seems your projects use too much memory. If you'll attach your project config that causes this I can have a look.

4
#4

Jordan

06.04.2021 03:32
Dear Mark, I'm using 131.612 now. I find that there are many problems in the memory management of this version, and there are no similar problems in previous versions. My computer's memory is 64g. I set “set maximum memory to 50g” in configuration / memory and restart the software. In actual operation, sqx takes up more than 55g of memory, sometimes it will directly fill up the memory. Sometimes the memory is not full, but sqx has no response, so I can only restart sqx.

I did turn off optimizations / don't store data for 3D charts in optimization profile when running. Because I need to save relevant data for analysis, I think my computer's CPU, memory and hard disk should be enough. This place seems to need to be optimized.

Yesterday, when I was using the custom project module, I set "set maximum memory to 50g". After running, several custom projects stopped automatically because they could only use 50g of memory, but the memory usage was not released. It was always above 60g, and the CPU usage reached more than 90%. Sqx did not respond, and I didn't know what sqx was running.

MF
#5

Mark Fric

07.04.2021 09:13

Status changed from Waiting for information to Refused

sorry, but this is nothing that can be solved by programming. 

Your setting uses the maximum of your memory, and when the memory is fully filled it wil bring many problems that you experience.

The version 131 has more effective parallelism for optimizations and retests, perhaps this is why you see the problems.

There are some fixes in the latest Dev 4 version, but I'm not sure they will solve your problem.


Please check the configuration of your projects to simply use less memory:

 - have less strategies into databank

- decrease number fo islands

- decrease maximum steps in optimization




Votes: 0

Drop files to upload

or

choose files

Max size: 5MB

Not allowed: exe, msi, application, reg, php, js, htaccess, htpasswd, gitignore

...
Wait please