Option - Delete strategies wit correlation > threshold

Voted for this task.

Absolutely needed if you mean correlation based on equity curve! Yes, please let us set a value from 0% to 100% similarity that will then clean-up the database and also make it an acceptance criterion, please (e.g. "if the equity curve is >98% correlated with another strategy in the database, don´t add it"). This will keep the database from overfilling with similar strategies towards the end of the genetic evolution. And it could be made even more smart, like: "if the equity curve is >98% correlated with another strategy in the database, check if the new strategy that is to be added has a higher fitness than the existing one, if it has higher fitness, add it and remove the existing one from the database". But that is most likely too much to ask for ;-)

Voted for this task.

imagine that you will need to compute the correl between every 2 members of databank - it will be very time consuming and you will have the problem which strategy to choose to throw away. What will be the filtering criteria? Throw away the worse strategy in the meaning of RDD, does this strategy more robust?

After you throw away one member, the correlation matrix could change, so you recalculate it?

So i am afraid that this filter in build process or in retester is not a good solution

I am doing this only on the final pack of strategies - put STR files to QA and manually deleted which has monthly correl higher than 0.5

the point is to have an option in retester to delete strategies which have correlation > threshold. not during bulidning 

I agree with hankeys. That filter could be time consuming and could delete the more robust strategy which is bad. I also preffer after I passed all the tests analyze the correlation in QA.

Careful because QA is not fully compatible with SQX strategies; there's an open bug

i understand - but which one to delete?

max correl is 0.71 between 1.23.88 and 5.16.77 - which one you delete? what will be the metric for it? RDD? higher RDD means better strategy?

OK - you delete 1.23.88 what next? do you need to recalculate the correl matrix? because it could change significantly after delete of 1 member

Voted for this task.

I can tell you that there must be a quick solution because another platform I use has exactly this and it takes less than a second to calculate for 100 strategies on 32 years of history data. So there must be a way. You also usually use that filter with really high values (99% correlation). This nicely avoid endless duplicates during genetic building on the other platform towards the end and seems to cost no time at all, even for large collections. No idea how they do it there though, but it works :-)

And of course, it should be an OPTION as usually, no one should be FORCED to use it if they don´t like the idea. For me personally, it is a crucial feature that makes me use the other platform a lot and even if it would take 10 seconds to calculate each time, I would not mind.

Voted for this task.

i am not talking here about that it couldnt be done - everything could be done - but only about that, it has some problems, which need to be settled, because this kind of filtering could lead to worse strategies in the meaning of their future performance.

So OK, if someone want to throw away some strategies, he need to tell how...

already for now the SQX could filter "too similar" strategies - i dont remeber it absolutely 100%, but it will take RDD and NetProfit with some light threshold and delete these duplicates and keep the one with higher RDD.

So i think 99%-100% correlation is settled.

I want to keep my pool of strategies as much as low correlated, i told here how i am doing - keep monthly correlation to max 0.5, but i do this only on final strategies from every pack (10+) and manually.

If i have in the final pool lets say 40 strats and make a correlation matrix and see that there is more values higher 0.5 and i dont want do it manually - i will set everything to portfolio master. Lets say, i will tell, that i want to keep only 10 strats max from this pack - i will run portfolio master 10-10, monthly correl max 0.5, rank RDD and keep the strategies from the portfolio with the max RDD

Long time i was working with QA ( 1 year) . You are right i did all the job in QA. Anyway Geektrader has good point too.

@hankeys:

>So OK, if someone want to throw away some strategies, he need to tell how...

We did, by the correlation filter mentioned in this feature request. Your method is nice, but complicated, I am happy to work with a simple balance line correlation filter instead (the one requested here) as I have good experience with it in the other platform that I am mainly using these days. If you have another idea, just open a new feature request, won´t hurt to have even more options. And again, no one has to use the filter if she/he doesn´t want.

Voted for this task.

@geektrader:

max correl is 0.71 between 1.23.88 and 5.16.77 - which one you delete? what will be the metric for it? RDD? higher RDD means better strategy?

from this example you want to throw which strategy? because you want to throw away only 1, right? but which one? i dont read here any solution for this...

Can you determine a relative fitness for each strategy based on a custom fitness that the user sets and then allow user to delete any highly coorelated strategies below a certain fitness score? It should go back in and delete only the lower performing strategies based on how the user would want it. 

Or just use portfolio masters functionality with the top x strategies below a certain correlation score when put together like @hankeys said. But strategies would first have to be weighed by some custom fitness so that you can only delete strategies with lower fitness scores.

something like this is planned - it will be a new module "Portfolio" where you can keep portfolio of your strategies, and then use Builder to generate new ones that will not correlate too much with your existing portfolio strategies.

We will get to it when we'll be free from most of the bugs, I expect in February- March.

Voted for this task.

Voted for this task.

Voted for this task.

Voted for this task.

Voted for this task.