All this used to be full on 100% utilization of CPU 100% of the time in previous versions. Only thing that changed that I can think of is the ability to split the IS and OS data granularity - You can now choose IS simulated and OS exact for example - previously the granularity choice applied across both the IS and OS components. Whether this has anything to do with it I don't know, but it was the first thing to spring to mind / gut feeling at least.
Attachment 1 GBPJPY H1 Simple V2 - WALK FORWARD MATRIX.cfx added
For WF Matrix I get 100% CPU saturation for the optimization part which is good. But for the computing runs part it is idle at 6% and no more. Each Run calculation takes around 25 minutes. Occasionally during the run calculations the application freezes for around 30 seconds and during this time processor use peaks at 69%, before unfreezing and the CPU dropping back to 6% as the run calculations continue. It does not 'move on' to the next run calculation when this happens, so I dont think it is 'looping' while at 6% then getting out of said loop for 30 seconds and higher CPU to calculate the calculate actual run before moving on to the next run calculation and looping at 6% etc. This was my first thought, but it doesn't appear consistent with the start and end of each run calculation.
Attached the config. Optimization part takes around 2 hours at 100% CPU usage. Run Calculations take 25 min each at 6% CPU usage.
I put several strategies in the results bank to be tested (WF Matrix).
The issues happens every time, but here is the curious thing...only for the first strategy it tests, subsequent strategies it tests concurrently in the same run complete much faster.
Regardless of the strategies, the first one takes approx. 2 Hours at 100% CPU for optimization, and run calculations 25 min each at 6% CPU.
All subsequent strategies in the same run take 30 minutes at 100% CPU for optimizations, and run calculations just 1 minute each at 6% CPU.
The flat-lining in the run part is still there, but its duration is massively reduced for subsequent strategies in the same run. The Peak up to 69% in the run part is also still there, but its duration is the same. The flat-lining either side experienced in the run part of the first strategy it tests is greatly reduced in subsequent strategies it tests. And the optimization component takes only a quarter of the time in subsequent strategies.
Hopefully enough info to root out whats going on and fix it. It would mean WF Matrix tests complete 10 times faster if this bug is solved!
I first posted about this back in January, and then again in February
https://roadmap.strategyquant.com/tasks/sq4_4094
https://roadmap.strategyquant.com/tasks/sq4_4024
Hopefully third time lucky with this thread!
It would mean the first WF Matrix test (strategy tested) would complete 10 times faster if this bug is solved!
Status changed from New to Duplicate
Let me address these things one by one:
1. WF use all cpu power for the first 30-60 seconds than drops to 5% till the end - it is because the optimization part is done in parallel, but the simulation (second) part is not - we have to improve it in the future. The reason is that you probably have Walk-Forward type configured as Simulated IS, Exact OOS, or Exact IS, Exact OOS.
With Exact OOS every OOS part in WF matrix is backtested using real tick data. There can be hundreds OOS parts in WF Matrix, and as of yet they are backtested in serial, not in parallel. But backtest itslef takes probably less time than preparation of the data for teh backtest - this has to be done in the first strategy, then it is reused, which leads to second point:
2. the first WF strategy test is slow - yes, but the reason is the preparation and memory-caching of the data for backtests for each OOS part. They have to be generated for every OOS part, and it takes some time. So the first strategy is slower because it includes these backtest data preparations.
I would highly recommend using Simulated IS, Simulated OOS - the difference in precision for timeframes from H1 up will be not that big, and the increase of speed will be significant, especially if you use tick data!
You can run Exact OOS later for the strategies that pass the rest of filtering.