<- Back to Tasks

[118] Still having serious "ran out of memory" issues

Whatever I try, SQX always runs out of memory for me after a few generations, despite having allocated 26 GB to SQX and even though there are just 2 strategies in the database yet (so this can´t take much memory). Additionally, I still get it at that 85% mark instead of 99% or 100% like it should be, even while I have clearly turned this 85% protection OFF in the options. Stopping at 85% with an allocation of 26GB means SQX will NEVER be able to use 4GB of the 26GB at all. Please remove that 85% stop, it´s totally pointless with large memory allocations and a waste of otherwise usable memory.

On top of that, the 85% stop occurred at around 18GB and you can see the heap memory limit (green) just allocated ~21GB out of the possible 26GB at this point. This all makes very few sense, to be honest.



Please see screenshots and I am sending my building settings via email as I do not want to share them completely in public.


Data used is GBPCAD 1986 -> 2018, H1, selected timeframe only, so about 200.000 bars. Even though you might not have this data, you should be able to recreate the issue with another timeframe (M1 or M5) as long as you are using at least 200.000 bars.


Right now SQX is still completely unusable for me if trying to use genetic evolution (fine in random mode though).


Thank you.

Attachments
memory02.png
memory02.png
(420.53 KiB)
memory01.png
memory01.png
(525.16 KiB)
  • Votes +2
  • Project StrategyQuant X
  • Type Bug
  • Status Fixed
  • Priority Normal

History

g
#1

geektrader

09.02.2019 11:14

Task created

g
#2

geektrader

09.02.2019 11:34

Description changed:

Whatever I try, SQX always runs out of memory for me after a few generations, despite having allocated 26 GB to SQX and even though there are just 2 strategies in the database yet (so this can´t take much memory). Additionally, I still get it at that 85% mark instead of 99% or 100% like it should be, even while I have clearly turned this 85% protection OFF in the options. Stopping at 85% with an allocation of 26GB means SQX will NEVER be able to use 4GB of the 26GB at all. Please remove that 85% stop, it´s totally pointless with large memory allocations and a waste of otherwise usable memory.

On top of that, the 85% stop occurred at around 18GB and you can see the heap memory limit (green) just allocated ~21GB out of the possible 26GB at this point. This all makes very few sense, to be honest.


Please see screenshots and I am sending my building settings via email as I do not want to share them completely in public.



Data used is GBPCAD 1986 -> 2018, H1, selected timeframe only, so about 200.000 bars. Even though you might not have this data, you should be able to recreate the issue with another timeframe (M1 or M5) as long as you are using at least 200.000 bars.


Right now SQX is still completely unusable for me if trying to use genetic evolution (fine in random mode though).


Thank you.

IH
#3

clonex / Ivan Hudec

09.02.2019 12:47
Voted for this task.
h
#4

hankeys

09.02.2019 13:12
did you see you strategies with 25000 trades? thats not normal
h
#5

hankeys

09.02.2019 13:19
8000 generations, 1024 for island? are you trying something?
g
#6

geektrader

09.02.2019 18:16
25000 trades is OK, it´s 32 years of history data, so not that much per year. Evolution settings are normal too, thats how I always used them - also makes no difference if just using 512. 3 Islands is normal though, others are using 10 or more. Running the same settings in build 115 gives no issues at all, barely uses > 11 GB of memory all the time. In 117 and 118 memory runs full in ~10 minutes in. Although it´s not really full, as that 85% protection (although I have DISABLED it), still kicks in and at 26GB total allocation leaves out 4GB that never can be used, which makes no real sense at all.
MF
#7

Mark Fric

28.02.2019 10:14

Status changed from New to Waiting for information

I think this could be fixed in B119, we fixed some memory issues that could have caused this.


Regaarding protection - in your case it seems it was not our protection, but a real OutOfMemory exception by Java that was caught and project was stopped because of that.


It would help if you'd attach your build settings, I tried to match your settings and memory consumption never crossed 9 GB in B119.

TB
#8

Tomas Brynda

28.02.2019 10:34

Status changed from Waiting for information to In progress

We already have the build settings from a private email. I will send it to you and check that memory limit
TB
#9

Tomas Brynda

28.02.2019 13:27

Status changed from In progress to Fixed

I have checked the inside memory control - it isn't applied when memory protection is turned off. The project was most probably stopped because of OutOfMemoryError in Java. 


From your screenshots I can see the project was stopped at 9:53 and the screenshot with error dialog was opened at 11:06. I think SQ ran out of memory, stopped the project and then some memory was released as SQ was idle since that time. You can provide a log file from that day if you want and I will check it out.


We have fixed some memory leaks in recent days and tried to let Build process and WF optimization run overnight to check the memory problems are solved. 

It looks good now, we haven't experienced any memory problems so far. SQ should be more stable after next build and should not consume so much memory.




g
#10

geektrader

28.02.2019 20:14
Thanks, that all sounds great! I do not have the log file anymore but will simply retest with build 119 once it arrives. Thanks again :-)
TB
#11

Tomas Brynda

28.02.2019 20:22
Ok, I believe it will be working well in the upcoming release. Thank you for your reports :)
MF
#12

Mark Fric

01.03.2019 12:50
thi sshoudl be fixed now
Votes: +2

Drop files to upload

or

choose files

Max size: 5MB

Not allowed: exe, msi, application, reg, php, js, htaccess, htpasswd, gitignore

...
Wait please