Optimizer WFM CPU usage low, no multi thread

seems WFM engine is different from BUILDER

no concurrent computations but rather sequential, right?

Voted for this task.

can you attach your optimizer config? Optimizer is parallel, there are some things that are sequentional. The CPU usage depends a lot on the config, once I'll be able to reproduce it with yours I can find a way to improve it.

b130. Optimizing all databank in a custom project. CPU ~ 25% for entire batch of 1400 strats. 32 threads. Optimize 100 iterations.

Looks like only 1 or a small number of strategies are being optimized simultaneously. I understand other people will have large number of iterations and smaller amount of strategies. It would be nice if we could preselect either one strategy per thread or one iteration per thread. Or even auto select by looking at total_strategies vs iteration ratio.

Voted for this task.

Voted for this task.

I do remember though even on my 16 thread machine, this operations was using very low CPU.

This one is not small, unlike other CPU related tasks which are effected only near the end of a task, this one is the entire task runs @ 30% CPU, a daily 2 hour task becomes 6 hours.

Voted for this task.

Voted for this task.

Voted for this task.

Still says you're waiting for information.... were you able to reproduce an issue with the cfx I attached above?

Voted for this task.

yes, I was able to reproduce it, I'm looking into it

bendx77 - the problem in your config is that you use only 100 iterations. Strategies from databank are optimized sequentionally for many reasons, and 100 max steps is not enough to make full use of all cores.

Using only 100 steps is a bad practice anyway, you should use at least 1000-5000 steps, otherwise the optimization will be quite radom if you have more than a few params.

I'm not sure if benalmador - original author of the task - had the same issue, if not please you or somebody else attach another optimizer config for me to look at.

Says who? I find 100 steps to be efficient for an improve strategy step before crosschecks are applied. More steps = opportunity to curve fit = more curve fit. (I'd probably be using the improver if it could improve  small changes to variables etc. as my other request.)  but I've found, even for WF and optimization purposes from limited testing 100 steps is the most likely to actually improve in hold out. I'd like to try 50 or 20 or 10 steps but I have a limitation from SQX that 100 is minimum. I can't be the only one who found it was best to stop early the mt4 or mt5 optimizations around 50-100 iterations years back before using SQX.

I can confirm adding more steps puts all the cores to use. It would be nice if that wasn't necessary. 

-Kevin Davy uses a couple of examples of 100 iterations pages 106 and 107 in Building Winning also a quote from page 76 in the book "Gently optimize whatever you do."

Maybe it can be a setting: 1 iteration per thread or 1 strategy per thread, then we can tune it for what's best for what we're doing and then you don't have to worry about all these cpu complaints? Or even automate: iterations > number of strategies then go 1 thread per iteration and vice versa.

using so little steps requires you to know what exactly you are doing. You can do it if you have just 2-3 parameters, and number of possible parameter combinations is small.

But if the number of possible parameter combinations is in thousands or tens of thousands then optimizing with max. 100 iterations is not much better than just randomly choosing any value from the range - it will not give you an overview how strategy behaves in the whole parameter space.

Because of the way it is implemented it is not so simple to configure it freely, it has soem implications that you don't see.

As I said, using just 100 steps is an extreme case.

If you really want to do it you can run several custom projects with Optimization task in each of them to fully use the CPU.

Thanks for the explanations and suggestion.

I thought more people would be interested in this but OK.

For what it's worth, optimizing with low iterations is a significant improvement nearly 100% of the time in sample and like I said is also significantly more likely to improve oos (this is so important!!!! Please test this for yourself!!!). The Comparison of a completely randomly selected iteration with the BEST one iteration selected out of 100 random is illogical because best out of 100 is almost always way better and choosing randomly would obviously not be an improvement at all. 

I get that a full analyses requires more iterations but for just trying some variables and selecting the best one and going with it OOS (OOS being the keyword) it is best to use fewer iterations to avoid curve fitting. Yes you can lower the optimization search space by increasing step size, shortening ranges and, as Mark says, reducing variables but, SQX can also do it very nicely by making the max iterations smaller and the benefits are similar (Hard to believe for some I know! Thankfully the miracle of SQX allows us to test these things very easily for ourselves so nobody has to take my word that it's good for OOS or Marks word that it's nearly just random. Results will vary depending on build method etc, test to find what's best in your own case)

When the context is proper OOS testing, 1000+ iteration fairs significantly WORSE than original strategies in most of my testing! I will be exploring running multiple projects to use the CPU.

Voted for this task.