SQX Cannot See more than 128 threads/logical processors

Voted for this task.

Voted for this task.

Hi Kim

What happen if you select custom number of cores and you enter 330 in the number of cores ?

Does it increase the number of cores ?

(On my side , I am using always this option, but I have less cores)

That was the first thing I tried. Unfortunately no go!

I am sure SQX team will find a solution shortly

Hi,

Just today I formatted my computer and loaded it with Kubuntu 22.04 and then I downloaded SQX which I notice is now version 139 (just released today).

Now... SQX sees all 384 processors but there are 2 problems....

1. The custom number of cores go up to max 200 only. Try typing in something more than 200 the number will come back to 200... And if you keep pressing the plus button it doesn't go beyond 200. 

2. Somehow even though now SQX in can see all the cores... But it is not using all of them... Back when I was running on Windows Server 2022 with 128 cores SQX would consume about 23%-30% of total CPU which makes sense because that's about 1/3 of the calls I have. However, with this Linux version of 139... Total CPU use is around 17% - 23%... And that is no difference in the speed of strategies generated.

FYI... Not sure what's going on.

See attached screenshots.

Also.... When I downloaded the new Linux version 139... I copied the user folder from the Windows version and replaced the user folder in the extracted download. Is that the right way to transfer the projects and data over?

I usually rename the extracted user folder to user1, then I would copy the previous windows user folder to the linux folder.

I do it like this windows system, but I never did it with linux. Can linux read windows files ?

It seems to be working so far. The formats of the files are SQX proprietary so Linux vs Windows shouldn't be a problem. Looks like copy the user folder is the way to go then. Thanks! Hope we can get the cores problem sorted out soon so I can test many different things and really understand my strategies.

Kim

This is very good that your reported it, now SQX team will take care of it. this is for sure.
Reporting each bug is very important for SQX.

In between , you can start a second instance of SQX, from a different folder in another hard drive,  to use the second part of your computer.

A bit more information in case it will help.

Please see the attached 3 screenshots... Here I am using the Linux version of 139. I am running 7 custom projects at the same time. I started with 1, then 2, then 3.... 

The CPU utilization and RAM use is capped at arond 50% and 125GB... I have DUAL Physical CPUs and 256GB RAM.... it seems just about HALF for both.

So I thinking this information might be useful.

I'm also testing to see if this would crash the system like the Windows 138 version.

Kim

Hello Luesak,

there's an easy solution for this, run multiple instances of SQX simultaneously.

It has several advantages, and SQX is optimized for it.

The CPU core usage limit depends on the OS and Java version. 

We'll take a look at it in the next version of sqx140, and try to improve CPU usage per instance.

Sincerely,

Tamas

Will I be sharing the same SQX database? If so that would be ok... if not that would be a real mess!

Kim

NO, the databases are not shared. You can manually copy the data from user folder to each instance separately (just copy the whole /user folder, it will work fine).

I would really like to run multiple instances of SQX... I think there's some kind of memory leak also... the longer I have SQX open... the slower it becomes and the memory stays locked at 125GB even though I now only run 1 project. 

Is there a config file that I can modify to have SQX point to a single database?

Thanks

Kim

Hi Kim

How much memory do you have ?

Yes in SQX folder , you have file called : StrategyQuantX_nocheck

you have a line :     option -Xmx480g

(I set up 480 Giga of ram, I leave Always 40 Gig for the rest of the system)

You can adjust it as well , for exemple if you have 260 Gig, you can set     option -Xmx200g 

(we talk about it on SQX discord.)

You can see my StrategyQuantX_nocheck attached and adjust your config too.

It's already like this

option -Djava.net.useSystemProxies=true

option -Djava.net.preferIPv4Stack=true

option -Djdk.tls.trustNameService=true

option -Xmx200g

option -XX:+UseParallelGC

But it's capping at 125G at least on Linux

Kim

Java option Xmx200g means that JAVA VM can allocate a maximum of 200GB. In your case, it seems that SQX doesn't require as much memory, that simple.

SQX has a built-in Java Heap space memory chart where you can see the memory usage - screenshot attached.

There's also a Memory Cleanup button for manually triggering the garbage collector and freeing up memory.

Is there a config file that I can modify to have SQX point to a single database?

No, this cannot be set. Databases cannot be shared among SQX instances.

You might be right about SQX only needing 125GB... but what I notice is that... if I run 3 projects at the same time RAM use is 125GB.

When I run 4 projects... it's still 125GB.... Then when I ran 7 projects... it's still 125GB... 

Not sure where the bottleneck is.

Based on what you're writing, SQX reserves 125GB at startup but doesn't utilize it during project execution. If it needs to allocate more memory, it reserves a maximum of 250GB.

Did you check Heap memory chart in SQX? 

• Heap memory chart – the current usage of heap memory and maximum heap size. The heap size increases when new objects of reference types are allocated, and decreases when they are garbage-collected. The maximum heap size is controlled by the -Xmx option.

Thanks for working with me on this! I'm thinking of how to run multiple instances of SQX and I'm testing it out. 

I went throught he user folder structure... it seems there's nothing in there that is really "DB"... I installed SQL Lite browser and saw that the custom projects are not in the DB anywhere... they are just really files. Having said that there are 2 things:

1. user/data: This contains the data for SQX to use... this one there is some information in user/data/data.db that needs to be manually synchronized between the instances ... meaning update one instance and copy it to the other instances

2. user/projects: There are no "db" or "config" files here... just a bunch of folders that self contain all the info regarding each custom project... so really this one is easy too... just copy the files over and they will appear on whatever instance we copy over... 

Can you confirm if my understanding is correct? I've partially tested it and it seems SQX works this way... if that's the case, I can continue to plan how I'm going to maximize the use of my computing resources.

As for the memory heap... mine looks like the attached... these 2 instances are running 1 project each... both configured to max 230GB... so it seems they both don't go over 100GB and use pretty much the same memory and also the memory is being realeased only when it reaches 95+GB??? Otherwise we should really see the red zone going up and down more (like yours). 

Kim

Further information... see screenshot... running multiple instances of SQX does not work. 

Config for each instance:
1. All but 1 CPU for UI

2. 75 GB Ram

As you can see... instance 2 and 3 are barely doing anything! And using very little RAM.

A bit more info... I hope all this info I'm sending is going to help with troubleshooting the problem!

I decided to puase SQX1 to see if the other 2 instances will use more CPU... to my surprise... they didn't! Just sat there with low CPU as if I was still running SQX1!

Kim

Kim,

On SQX discord , we talked about a software to manage multiple cores for SQX . Do you use it ?

This software is suppose to make all the cores available for SQX , did you try it ?

Tried it. The software is called Processor Lasso. The function now is called CPU Sets. They removed the original funciton that was used pre Windows 2022... because Windows 2022 is supposed to automatically utilize all the processor groups. However, SQX 138/139 windows version has a problem in that it only sees 1 processor group per pyhiscal CPU. SQX team is fixing. So because it only sees that many, even if you use CPU sets (which I did), SQX will not "spawn" new threads or do more than it "sees". At least that's from my observation and suspicion as a programmer for 25 years.

So now... the best thing to do is run 1 instance of SQX and give it ALL - 1 the CPUs and as much RAM as you can. This is the fastest way right now. Until SQX team makes a fix to use all CPUs more effectively. They mentioned upgrading to Java 22 from Java 15 might do the trick. Not sure. 

Kim

Got new potentially critical observation....

CPU utilization is directly related to two things 

1. # of Blocks - The more blocks the lower the CPU usage 

2. # Islands - The more Islands the higher the CPU usage 

So if you choose all blocks and you have 10 islands... CPU will stand around 10% 

But if you choose 40 blocks and you have 10 islands... CPU will be around 23% 

But if you have all blocks and 100 Islands CPU shoots up to 100% using all 385 cores and sustains at around 50%

Hope this helps

Thanks for the update! We'll dig deeper in build SQX140.

Yes, you can use program Processor Lasso to utilize all cores on your PC. 

This solution is also mentioned on SQX web here

https://strategyquant.com/blog/efficient-memory-management-and-fixing-stability-issues-for-strategyquant-x-sqx/