Only one processor socket is recognized and used - logical core limit?

I believe Java should be able to use multiprocessor architecture. Is this configuration of yours a physical machine or some virtual machine? 

I think the problems could be related to OS or BIOS drivers, please check more in this link: https://stackoverflow.com/questions/35667557/java-process-not-accessing-all-cores-in-windows

and in the link in the first answer.

As for memory - it doesn't depend on number of cores that much, it is not that 1 core uses 4 GB of memory.

It depends more on how many strategies you want to keep in databank, and I don't think it makes much sense to keep tens of thousands strategies there.

You should use advanced filtering in SQ so that it automatically throws away "bad" strategies, instead of storing 100k strategies. SQ was not build to handle that many strategies in databank.

So 32 - 64 GB of memory should be enough for most scenarios.

Reading around, windows groups virtual cores into groups of 64 maximum. Once 64 is reached, and second group is created for the remaining cores and so on. You can see this in the affinity settings for the exe process. In the case of SQ4 only one group of processors are assigned, the rest are not. You can assign both groups, and thus all cores to a process, but in the case of the SQ4 process, this is not possible - if you assign both processor groups to the process it always immediately reverts back to one group only.

It it possible to allow SQ4 to use more than one processor group? I am sure its not that simple.

Still leaves my original question unanswered - can SQ4 use more than 64 virtual cores across several dual core server racks? If yes, then its just an OS or BIOS issue my end and I will pay a techie to sort out. If it no, then it would be great to have confirmation so I don't waste money on several additional servers that cant be utilised :)

N

I confirm that there is no limitation in SQ, it can use any number of cores as long as Java recognizes them all.

You can try two more things:

1. add the following line to StrategyQuntX.config file:

option -XX:+UseNUMA

then start SQ. This should switch on NUMA mode. I'm not sure it will help with processor groups, but it might.

2. try the newest Java 10 JDK. Simply download Java SE JDK from http://www.oracle.com/technetwork/java/javase/downloads/index.html

and replace the folder {SQ}/j64 with this JDK folder.

This will make SQ use Java 10, perhaps they made some enhancements or fixes for the multi-CPU architecture.

SQ X uses Java 8 by default. We still haven't switched to Java 10, because it requires more tuning, SQ on J8 is faster than J10 in our tests so far.

3. as the original link stated, it could be a bios issue:

https://support.hpe.com/hpsc/doc/public/display?sp4ts.oid=5379860&docId=emr_na-c04650594&docLocale=en_US

and depends on hardware vendor.

All in all, it is something that might be not solvable - we cannot change JVM implememtation, or drivers if they are not provided by vendors.

Thanks Mark.

I tried both of those, independently and together, and that didn’t work unfortunately.

While it could be the BIOS I still feel this is unlikely as I am running into the same issue on multiple different machine configurations (Google Compute, AWS) on multiple setups and platforms (Skylake, Haswell, Broadwell, AMD). The commonality here I would suggest is not the BIOS, but the virtual (VM) environment, if indeed there is one.

Reading around it appears that more threads/processor groups need to be explicitly set in the options. Where exactly these are options are defined I am not sure, some potential candidates that have come up:

-XX:CICompilerCount=threads

-XX:ConcGCThreads=threads

-XX:ParallelGCThreads=threads

-XX:ThreadStackSize

-XX:+AggressiveHeap

I have no clue to be honest, until today allI knew about Java was that it had something to do with Oracle :)

I also read that you can set the affinity mask of the java process to utilise all available cores and processors, and again attach these setting to a specific program executable so they will be propagated to the application.

I looks like a setting somewhere that tells java to use all available processors, and these are then reflected in the processor number in SQ4

These guys have the opposite problem and want to restrict the number of CPU’s used by a java application, and are selling their code…whats interesting is they  have managed to define the number of CPU’s used by the java service (cpu_affinity.<n>)…which at the very least suggests that it can be defined with an option/setting in the Java process.

https://wrapper.tanukisoftware.com/doc/english/qna-processor-affinity.html

This is a physical core limitation. If you are running a machine with 60 physical cores like me, then SQ4 will only use half of these and run at 50% of processor utilisation - this is because the BIOS presents the logical (not physical) cores to the OS.There is a way around this on some machines in the BIOS settings - on my Dell machine I disabled logical processors in the BIOS - this then presents only the physical cores to the OS. SQ4 will then see 60 physical cores (instead of only 60 logical cores) and use these, and the hyper-threading to 120 logical cores happens out of sight of the OS. Changing this setting will have no impact on your benchmark score in SQ4, this will remain the same (inaccurate), but if you 'start building with default' and let it run for 3 minutes to get a stable average and look at the strategies per hour value you will see it double :) You will also see task manager reporting 100% processor usage versus 50% previously. I now get 1,200,000 strategies per hour versus 480,000 per hour afterthis BIOS setting change.

Conclusion: don't waste money on a server with more than 60 physical cores as they will not be used, and if you buy at this 60 core limit, make sure the server has the BIOS setting to disable logical cores, otherwise SQ4 will only use 50% of your processor power. SQ4 will never 'see' more than 60 cores - be they physical or logical. If you want more than 60 cores, then you will have to buy a second server and wait for SQ4 to release the GRID COMPUTING update to take advantage of them over a local network.