CPU usage drops rapidly during Gen EVO

It goes to nothing nothing until next randome gen when it spikes up again this without any crosschecks added.
Attachments
No attachments
  • Votes 0
  • Project StrategyQuant X
  • Type Bug
  • Status Fixed
  • Priority Normal

History

m
#1

mabi

23.06.2018 04:47

Task created

m
#2

mabi

23.06.2018 04:59

Attachment CPU.jpg added

m
#3

mabi

23.06.2018 05:12
Seems you can keep CPU usage up with many Islands and larger population size plus harder filter rules which forces it to randomly seek for new strategies to fill up the Islands I suppose .
MF
#4

Mark Fric

26.06.2018 10:12

Status changed from New to Waiting for information

Attachment cores.png added

You are right that when you use too little islands and too small population then in later generations there might be not that many "new" strategies to be tested.

How many islands and how big population do you use?


Also, from your screenshot it looks like it doesn't use your first processor. Is it still like this even if you use 40 islands and 100 population?


I also have double Xeon PC, and it uses all cores in my case.


Can you also check log? There should be a line:

10:01:51.758 [main] INFO  c.s.g.utils.CoreUsagesEvaluator - Using percentage of cores:100%. Total cores:48. Cores for usage:48

10:01:51.758 [main] INFO  c.s.g.utils.CoreUsagesEvaluator - Total cores available: 48, using: 48


showing how many cores will be used in parallel processing. What does it say on your computer?

m
#5

mabi

27.06.2018 04:00
I am unfourtunately on bussines trip on (mobile )until 1 July and have a problem checking this until then. SQ4 seems to not be Numa aware. If you can set bios to Uma mode it would work but i cant. When looking at affinity SQ4 can only be set to use one of the numa nodes and all hyperthread cores in this. Anyway i have no problem running 2 sessions of SQ4 and set affinity on each to use diffrent Numa nodes which maybe is more effective anyway. I think below post and answers provide the answer to the problem.


windows/en-US/9589087d-9036-4b39-bfce-059aed7b2abf/windows-allow-only-a-maximum-of-36-logical-processors-per-process-out-of-total-of-72-?forum=winserver8

m
#6

mabi

27.06.2018 04:28
Seems below link with more recent comments in the end is more informative of the problem with windows grouping of sytems with more then 64 threads.


https://social.msdn.microsoft.com/Forums/windowsdesktop/en-US/e0f49cda-4cbf-4da0-918a-0568271edb9f/process-affinity-on-a-system-with-128-processors?forum=wdk


MF
#7

Mark Fric

27.06.2018 14:22
I'm not sure that this is related to NUMA, NUMA is about memory, it shouldn't be a cause why Java is not using the second processor.


SQ4 uses standard Java 8, which should have no problem using multiple processors.


There is a possibilty to add a specific command to StrategyQuant4.config file:

option -XX:+UseNUMA


thyat will turn NUMA support on, but I'm not sure it will help.  


It would help me if you can end me these few lines from your log so I can see how many cores SQ4 really see.


Perhaps upgrading motherboard driver would also fix it.

m
#8

mabi

29.06.2018 08:01
Read some more this morning. It seems Windows allows max 64 threads and if you have more it will create a new group. SQ only see and uses one group.This numa support only has to do with memory handling. Anyway i will try it tomorrow when i am back. It is probably solved by creating another worker as you plan with grid computing
MF
#9

Mark Fric

04.07.2018 11:37

Status changed from Waiting for information to Fixed

I'm not sure if it is fixed, but it is most probably related to drivers - motherboard or CPU. 


There is no reason why Java would use only one processor, except that it doesn't see it - and that is a drivers issue.


Votes: 0

Drop files to upload

or

choose files

Max size: 5MB

Not allowed: exe, msi, application, reg, php, js, htaccess, htpasswd, gitignore

...
Wait please