132: support processor groups

I have noticed the SQX does not make use of CPU threads that span across multiple processor groups. I’m asking for you to add support for processor groups which will allow me to make use of my 64c/128t CPU (AMD Threadripper 3990X).


 

For a detailed explanation of processor groups look here:

Link 1) https://docs.microsoft.com/en-us/windows/win32/procthread/processor-groups#:~:text=Support%20for%20systems%20that%20have,are%20numbered%20starting%20with%200.&text=The%20operating%20system%20minimizes%20the%20number%20of%20groups%20in%20a%20system.

Link 2) https://techgage.com/article/microsoft-processor-groups-beyond-64-thread/


 

When SQX (more specifically, the javaw.exe process) is loaded, Windows assigns the process to the processor group with the lowest load. Now, by default, applications can only make use of threads that are contained within the same processor group. For example, since I have 128 threads I have 2 processor groups each with 64 threads. This means that after the javaw.exe process has been assigned to a random processor group, it can only utilise 64 (50% in my case) threads because that’s the number of threads assigned to that process group. Effectively, I can only use 50% of my total CPU power in a single SQX instance. This is a real issue and fixing this would result in a huge performance improvement.


 

A few things that I have tried:

  • Disabling hyperthreading. This does not resolve this issue, it simply ignores it. Performance is halved based on SQX benchmarks I have made myself.

  • Process Lasso. Third party software cannot solve the underlying issue. It must be solved internally by your developers.

  • Playing with the CPU settings inside of SQX (manually assigned CPU threads to 128 didn’t allow SQX to use more than 64 groups because it is not looking outside of the processor group it’s assigned to).


 

I have attached 2 pictures which highlight SQX’s CPU usage whilst running a random generation strategy builder. Again, to be clear, this is not an issue with my personal SQX setup, or an issue regarding my windows version (all Windows 10 versions – Pro, Workstation, Server) split threads into processor groups of maximum 64 threads each. SQX needs to be updated and be made “processor group aware” to allow users to utilise all CPU threads they have available. Without this, SQX is simply not making use of the hardware it’s running on (excluding users who have less than 64 threads – in this case it’s fine since there is only 1 processor group).


 

If you need any additional information please let me know. I’m happy to be a “beta tester” for any solutions you would like to test.


 

Many Thanks and Best Regards,


 

Attachments
image-0.png
(192.50 KiB)
image-1.png
(20.07 KiB)
  • Votes +6
  • Project StrategyQuant X
  • Type Bug
  • Status New
  • Priority Normal

History

TV
#1

Tomas Vanek

23.10.2020 18:56

Task created

m
#2

mabi

26.10.2020 12:05
Voted for this task.
BS
#3

BobS

27.10.2020 14:45
Voted for this task.
N
#4

nathan

28.10.2020 09:54

Attachment Task Manager.jpg added

Attachment Task Manager.jpg deleted

Attachment Task Manager.jpg added

Task Manager.jpg
(167.10 KiB)
I agree, this would be nice, but if I remember correctly I think its a limitation of Java.


I'm running a server with 60 cores and had the same issue - 50% utilization.


I got around it by disabling logical cores (hyper threading) at the BIOS level. After that task manager showed 60 cores and 60 logical processors and 100% utilization in build. Before adjusting at the BIOS task manger showed 60 cores and 120 logical cores with 50% utilization.

WH
#5

Stormin_Norman

11.11.2020 10:24
"I got around it by disabling logical cores (hyper threading) at the BIOS level. After that task manager showed 60 cores and 60 logical processors and 100% utilization in build. Before adjusting at the BIOS task manger showed 60 cores and 120 logical cores with 50% utilization."


I just wanted to make it clear that this does not "get around the problem". You're simply avoiding it and missing out on a huge amount of processing power.

g
#6

geektrader

11.11.2020 17:23
Voted for this task.
N
#7

nathan

12.11.2020 11:21

Stormin_Norman

 1 day ago
"I got around it by disabling logical cores (hyper threading) at the BIOS level. After that task manager showed 60 cores and 60 logical processors and 100% utilization in build. Before adjusting at the BIOS task manger showed 60 cores and 120 logical cores with 50% utilization."


I just wanted to make it clear that this does not "get around the problem". You're simply avoiding it and missing out on a huge amount of processing power.



How so? The physical processors are 100% utilized? If I compare it to my 3 other servers with16 cores each, but with hyper threading on on these servers, the strategies generated per hour using identical projects is almost 4 times greater on the 60 physical core server with hyperthreading off, than on the 16 physical core server with hyperthreading on. If am missing something please let me know, would love some more computing power!

k
#8

Karish

11.05.2021 03:52
Voted for this task.
KB
#9

kbtech

08.07.2021 18:22
Voted for this task.
MW
#10

Goon

09.07.2021 03:50
Voted for this task.

Votes: +6

Drop files to upload

or

choose files

Max size: 5MB

Not allowed: exe, msi, application, reg, php, js, htaccess, htpasswd, gitignore

...
Wait please