Memory usage & speed issues

hi,
some issues with performances and RAM usage.
I start builder. for about 5-10 hours it generate about 500.000 strategy witth current setting.
then when I have about 15.000 strategies stored on databank it start to slow down. At this point ram consumpton is about 15GB and strategy per hours go towards 400.000 per hours.
than after 20 hours I have about 27.000 strategies on databank. RAM consumption is about 35GB and it generates about 380.000 strategies per hour. I clear log but nothing change.
I know SQ generated a lot more "good strategies" but it replaced with better one or just some of them are similar.
Some questions:
- why the "generation -speed" slow down also if memory is not fully used?
- with 15.000 strategies on databank initially RAM consumption is about 15GB. than with 25.000 is 37GB. it does not make sense (it is not proportional). maybe it is because it stores also replace strategies and similar ones. is there a way to clear some RAM? or use SSD?
- as you can see from screenshot it does make "auto-sink" of strategies every 4 hours. is it the cause of RAM huge increase utilization and speed reduction?

I attached several screenshots so you can understand better.
please le me know what can I do because with 64GB of ram in that situation I cannot have more than 30K  strategies on build databank and I'm sure it is possible do do better with such bug ram.

thanks
Attachments
image 3.png
(358.25 KiB)
image 4.png
(380.47 KiB)
image 2.png
(244.92 KiB)
image 0.png
(360.86 KiB)
image 1.png
(305.23 KiB)
  • Votes 0
  • Project StrategyQuant X
  • Type Bug
  • Status Refused
  • Priority Normal

History

TM
#1

tmatejka.

17.07.2019 14:27

Task created

MF
#2

Mark Fric

18.07.2019 14:59

Status changed from New to Refused

I'm sorry, but we are not sizing the program to handle tens of thusands of strategies in databank.


Even if you have a lot of RAM, Java has its issues when a big RAM is used, and it is not only problem of memory.

Data in databanks, etc. will be slow to compute when you'll have 30.000 records there.


The recomendation is to use maximum few thousands, maybe up to 10k records in databank.


It doesn't make a big sense to generate 30.000 strategies anyway. You should use retests and filtering, maybe custom projects flow to evaluate the strategies regularly in batches, so that the number of strategies doesn't grow too big.


Votes: 0

Drop files to upload

or

choose files

Max size: 5MB

Not allowed: exe, msi, application, reg, php, js, htaccess, htpasswd, gitignore

...
Wait please