WFM not using all cpu power (cores)

Voted for this task.

Reported exactly the same bug for the last couple of releases. Around 114 or 115 (I cant remember when exactly) WF would use 100% of CPU resources, since then no longer - only around 5%-15%.

Voted for this task.

What CPU(s) are you using?

Intel Xeons 

Intel Xeons E7-4890 V2

Voted for this task.

Voted for this task.

Voted for this task.

Voted for this task.

In my case it's 100% CPU until it starts computing the walk forward matrix, only 100% whie doing the optimization part.

For me it happens on both WF and WF Matrix. On the WF it is erratic, jumping up and down between 3% and 96%, rarely leveling off in the upper numbers, before falling back down again, the flat-lining at around 4% for a minute or so, then erratically building and falling again, then flat-lining again.  This also describes the behavior of the optimization part of the WF matrix, then the final matrix calculations at then end of the WF Matrix take forever, far far longer than the optimization part, with the CPU flat-lined the entire time at around 3%.

All this used to be full on 100% utilization of CPU 100% of the time in previous versions. Only thing that changed that I can think of is the ability to split the IS and OS data granularity - You can now choose IS simulated and OS exact for example - previously the granularity choice applied across both the IS and OS components. Whether this has anything to do with it I don't know, but it was the first thing to spring to mind / gut feeling at least.

Adding to the above comment after watching it this morning again, it is erratic as described for WF. 

For WF Matrix I get 100% CPU saturation for the optimization part which is good. But for the computing runs part it is idle at 6% and no more. Each Run calculation takes around 25 minutes. Occasionally during the run calculations the application freezes for around 30 seconds and during this time processor use peaks at 69%, before unfreezing and the CPU dropping back to 6% as the run calculations continue. It does not 'move on' to the next run calculation when this happens, so I dont think it is 'looping' while at 6% then getting out of said loop for 30 seconds and higher CPU to calculate the calculate actual run before moving on to the next run calculation and looping at 6% etc. This was my first thought, but it doesn't appear consistent with the start and end of each run calculation.

Attached the config. Optimization part takes around 2 hours at 100% CPU usage. Run Calculations take 25 min each at 6% CPU usage.

The SAD is that this bug is not assigned still, like it is not important to catch out all the power we can use to accomplish such a heavy task for filtering the strategies.

The config is for the Optimize task block (not the Retest task block)

I ran some more tests.

I put several strategies in the results bank to be tested (WF Matrix).

The issues happens every time, but here is the curious thing...only for the first strategy it tests, subsequent strategies it tests concurrently in the same run complete much faster.

Regardless of the strategies, the first one takes approx. 2 Hours at 100% CPU for optimization, and run calculations 25 min each at 6% CPU.

All subsequent strategies in the same run take 30 minutes at 100% CPU for optimizations, and run calculations just 1 minute each at 6% CPU.

The flat-lining in the run part is still there, but its duration is massively reduced for subsequent strategies in the same run. The Peak up to 69% in the run part is also still there, but its duration is the same. The flat-lining either side experienced in the run part of the first strategy it tests is greatly reduced in subsequent strategies it tests. And the optimization component takes only a quarter of the time in subsequent strategies.

Hopefully enough info to root out whats going on and fix it. It would mean WF Matrix tests complete 10 times faster if this bug is solved!

@loonly

I first posted about this back in January, and then again in February

https://roadmap.strategyquant.com/tasks/sq4_4094

https://roadmap.strategyquant.com/tasks/sq4_4024

Hopefully third time lucky with this thread!

CORRECTION 

It would mean the first WF Matrix test (strategy tested) would complete 10 times faster if this bug is solved!

Voted for this task.

ok, this is not really a bug, more an optional feature where it can be partially improved.

Let me address these things one by one:

1. WF use all cpu power for the first 30-60 seconds than drops to 5% till the end - it is because the optimization part is done in parallel, but the simulation (second) part is not - we have to improve it in the future. The reason is that you probably have Walk-Forward type configured as Simulated IS, Exact OOS, or Exact IS, Exact OOS.  

With Exact OOS every OOS part in WF matrix is backtested using real tick data. There can be hundreds OOS parts in WF Matrix, and as of yet they are backtested in serial, not in parallel. But backtest itslef takes probably less time than preparation of the data for teh backtest - this has to be done in the first strategy, then it is reused, which leads to second point:

2. the first WF strategy test is slow - yes, but the reason is the preparation and memory-caching of the data for backtests for each OOS part. They have to be generated for every OOS part, and it takes some time. So the first strategy is slower because it includes these backtest data preparations.

I would highly recommend using Simulated IS, Simulated OOS - the difference in precision for timeframes from H1 up will be not that big, and the increase of speed will be significant, especially if you use tick data!

You can run Exact OOS later for the strategies that pass the rest of filtering.

Understood, makes sense now :)