Severe CPU under-utilisation in WFM matrices - AMD Threadripper 3990X, with 64 cores


I am using the Linux version of SQX in order to bypass the Windows 64-core processor group problem, as I am using 128 logical processors. In most functions of SQX that I have tested (Build, Monte Carlo, SPP etc), the CPU cores are used to their full potential.

However, when running WFM robustness tests, on the 'Running backtest for WF Simulation' sections, these are not parallelized correctly, and only use a fraction of the CPU potential. This is most evident when running large matrices, and a greater amount of time is spent running each set of backtests. This is a huge waste of resources as for those that like to run large WFM matrices for several days, the CPU usage is only at 25-30% and takes much longer.

Considering that for other functions of SQX such as building strategies tasks, full CPU utilization is achieved, there should be a way to parallelize the large number of backtests for the WF simulations in order to more efficiently use the CPU (and save hours of strategy generation time).


The processor is an AMD Threadripper 3990X, with 64 cores (128 logical processors with SMT enabled, as it is here).
I am using the 'htop' command in Linux to visually track CPU utilization for different tasks in SQX.

Files attached:
1. Custom Project file used in the demonstrations
2. Screenshot of my CPU config within SQX
3. Screen recording monitoring the build strategies task (full utilization)
4. Screen recording monitoring the MC history task (full utilization)
5. Screen recording monitoring the SPP task (full utilization)
6. Screen recording monitoring a large WFM (UNDER-UTILIZATION)
7. Screen recording monitoring a small WFM (UNDER-UTILIZATION) - but not as obvious of a reduction as the large WFM due to the decreased number of backtests)







Processor architecture, in case it helps at all:

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         43 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  128
  On-line CPU(s) list:   0-127
Vendor ID:               AuthenticAMD
  Model name:            AMD Ryzen Threadripper 3990X 64-Core Processor
    CPU family:          23
    Model:               49
    Thread(s) per core:  2
    Core(s) per socket:  64
    Socket(s):           1
    Stepping:            0
    Frequency boost:     enabled
    CPU max MHz:         4375.4878
    CPU min MHz:         2200.0000

Attachments
SPP Full Utilization.mp4
(2.45 MiB)
WFM Large CPU UNDER-UTILIZATION.mp4
(4.74 MiB)
Build Strategy full utilization.mp4
(1.83 MiB)
Example WFM Project.cfx
(79.07 KiB)
WFM Small CPU UNDER-UTILIZATION.mp4
(4.40 MiB)
MC History Full Utilization.mp4
(1.59 MiB)
SQX CPU Config.png
(39.70 KiB)
  • Votes +5
  • Project StrategyQuant X
  • Type Bug
  • Status New
  • Priority Normal

History

MW
#1

Goon

11.07.2021 15:29

Task created

MW
#2

Goon

11.07.2021 15:34
Voted for this task.
m
#3

Martin

11.07.2021 16:14
Voted for this task.
JH
#4

Jabezz

12.07.2021 04:03
Voted for this task.
AA
#5

Alex

09.09.2022 04:42
Voted for this task.
TT
#6

Tamas

12.09.2022 06:18

Subject changed from Severe CPU under-utilisation in WFM matrices to Severe CPU under-utilisation in WFM matrices - AMD Threadripper 3990X, with 64 cores

E
#7

Emmanuel

05.04.2023 15:01
Voted for this task.

Votes: +5

Drop files to upload

or

choose files

Max size: 5MB

Not allowed: exe, msi, application, reg, php, js, htaccess, htpasswd, gitignore

...
Wait please