RAM ussage

Voted for this task.

but this is how Java Garbage collector works. It doesn't free memory when there is enough of it.

What is the problem with this? If you don't want SQ to use that much memory set it to use less RAM.

Voted for this task.

Voted for this task.

Exactly as Mark says, SQX cannot influence this, as the JavaVM is designed exactly like this. It will NOT free up any memory (perform garbage collection) if the memory you´ve allocated to it it is not already FULLY used. ONLY THEN it will perform it´s garbage collection and free up memory. So if you allocate 128GB to it, it will first fill up the 128GB, then perform a garbage collection, which could indeed free up 100GB of RAM. As garbage collection is still a very "costy" thing, it avoids it until not all of the allocated RAM is fully filled. If you want it to use less memory, you have to assign it less memory, then it will start to perform garbage collections as soon as that´s filled up and will most likely also work just fine with the amount you gave it. But the overall speed will be slower, as more garbage collection will be performed during your strategy generating. Overall I totally get what you are looking for, but this is one of the downsides of Java compared to C++ etc. and as long as they do not switch to another programming language, this is exactly how it will work RAM management wise.

problem is simple - even if i have 128 GB of RAM i can get pretty quickly to the point, when SQX crashed and if i restart its using only small portion of ram with clean run and with the same amount of strats before crash

Voted for this task.

Voted for this task.

@ hankeys, Just a hunch, do you have a StrategyQuantX_nocheckl4j.ini file? It may be overriding your garbage collector / RAM settings. You may want to delete it and try different garbage collector / RAM settings inside sqx.

Fresh restarts with all same strategies but less RAM usage than before a crash is normal but crashing is not.

@hankey: as mentioned, JavaVM will always first fill up the RAM you assign to it before performing any garbage collection (freeing up RAM), that is how java works and even if the coders of SQX would like to do it different, they cannot, because java does not allow any memory management, which is the good and the bad at the same time.

As for the crashes, that is not OK of course. Which JavaVM are you using? Try GraalVM EE and see if it still crashes. If so, did you test your RAM? Try this one for 24 hours: https://www.karhusoftware.com/ramtest/ it is just 10 Euros and has found any memory problem I ever had. Just like you, I am also running on 128GB RAM and 24 hours a day, without any crashes in months.

hankeys can you attach your project config that leads to crash? I'll investigate it

you can try for example this evaluation project - https://mega.nz/file/GFARjAbQ#GM8BSZFPdHGaVYbSbWW9_qIwpcxfVwBrlynJhat5E8M

ok, I was able to reproduce the crash. But it is a bad configuration, you first generate 10.000 strategies, and then you run a series of additional retests, including a few Monte Carlos with 200 simulations.

I think these Monte Carlos are responsible for the crash. On my computer the 10k strategies use around 10-15 GB of RAM.

With 200 MC simulations for each strategy it will be almost 200x that - well, not really, MC results are highly optimized so that they don't use as much memory as standard backtest but still they could use 5-10x more memory.

So the recommendation is to lower the number of strategies generated in the first step from 10k to 1k, OR apply better filtering so that only a much smaller portion of them gets to the MC test tasks.

yes could be, but with the retest task i am deleting the mc tests for the strategies and if this task is completed ram usage will not go down and remain the same...this is for me not logical

and if i change the project, so i dont save also failed strats and keep only passed ones the problem with big ram usage remains also

Java tying up memory that you have assigned to it (even if not needed by Java in that exact moment) should not be a problem in and of itself. Java should free the memory some point before it crashes if it can. It does not waste the resources required to keep the lowest memory possible, instead it only frees up memory at the last moment if needed and if it can. More efficient that way. Not an SQX thing it's a Java thing.

Yes it seems weird when you are used to other applications which free up ram for you more eagerly but if you read this you will see that it is in fact Logical:
https://www.geekyhacker.com/2019/01/04/jvm-does-not-release-memory/

That is the big downside of Java, as I said. However, try the

-XX:-​ShrinkHeapInSteps

option. It reduced the RAM usage during long runs a little bit, while having no impact on the Performance here.

when i ran the evaluation project above with 32 gig ram it crashed SQX. When i ran it with 64 it worked fine using up to 49 gig of ram on same putor. Then i used a thridparty mem cleaner and still Heap says it has allocated 49 gig of ram but Taskmanager now only show 21 gig. . To complicate things abit .. hehe

Voted for this task.