Similar Strategy Remover

Up, please i hope that you implement this feature in the next 109 build, make happy a very active member of this comunity ;-)

As someone who sorts through code for hundreds of different strategies, this would be great as code sorting takes up the majority of my time.

Most practical way to do this is sort by the entry condition in the code and replace strategies  in the databank with the same entry condition that have a higher fitness/net profit (user configurable replacement filter). any strategies with the same entry condition but a lower fitness (or user configurable filter) would not be added.

This would require code that compares what's in the current generation to what's in the databank already so I didn't think the feature itself would be simple and wouldn't expect this for a while. However, if it's practical I am definitely all for this

this is very hard to say - what are the duplicates - because you can have str. with the same entry building block, but in the filtering blocks could be very different conditions - which means, that the strategy could be totally different

even slightly different SL, TP, trailing stops could lead to a different strategy

my thoughts about remove duplicities is something like this

take 3 major values that could lead to the conclusion that its a duplicity

net profit

number of trades

ret/DD ration

set the threshold +-5% and if you have in the databank strategy with the same values +- threshold, keep the only one with the highest ret/DD

imagine these final strs, they are all STOP ones, with only 4 entry building blocks

HA - heiken ashi

HH - highest

LD - low daily

HD - high daily

but they are not similar at all and are not correlated, because fo the filtering conditions, because of the moneymanagement they use, etc.

after some filtering new strs from 108, i found 100% similar strs in databank - its across islands and generations - there must be some kind of bug....

in the old task we have discussed about it, and i think that there is some "automatical dismissal" done, but in 108 i cant see it

very simply, take the trade time series of the generated strategy and gnerate the correlation with trade time series of every strategy inside the databank, problem solved, in this manner we are statistically sure that are correlated/uncorrelated. By the way this is how the correlation of the strategies inside a portfolio is calculated inside SQ so it just exist, they must apply this to the generation process. Is a little bit more difficult than other simplest problem, but they are a big team of developer, they can do it, i trust him.

Yes hankeys is a very serious problem, like you showed it is evident, too much... And trust me this problem afflicts also the random generation.
The databanks that you showed is one strategy, the other are the same but "optimized" so with little difference in the period, or little pip difference in take profit and stop loss...

there already should be filtering of same strategies, I'll check if there isn;t some problem.

But really, it is not so simple to filter similar strategies. We shouldn't do it by source code, because change in just one parameter could mean very different results.

Computing correlation is possible, it would be extremely slow. Imagine you already have 1000 strategies in databank and you should compute correlation with all of them when you are adding new strategy.

What I see as a possible option is to take a "fingerprint" using some of the stats values like what hankeys suggested:

net profit

number of trades

ret/DD ration

and use 5% threshold. I think these three numbers are a good approximation of the real equity curve.

yes, thats what i am talking about - but make it settable - true/false

I think that here is where a gpu can be really used, a dedicated gpu only to calculate correlations, this makes the source code a little bit complex, but the results is great, really, a very scorrelated databank, so a very scorrelated portfolio is more makeable. If you don't have a gpu developer I propose myself, I would be honored to write the gpu kernel, I know OpenCL and CUDA. 

Basically, SQ generate strategies and put the good one (for all the dismissal filter) inside the gpu memory, instead to put it inside the dB directly, and a dedicated source code, so totally detached from SQ, (but triggered from SQ) invoke the gpu kernel to make a reduction and generate the correlation matrix. The strategies scorrelated come back inside the databank at the end of the generations. 

I have just created some similar, for SQ 3.8.2 but it works in post production, it would be more elegant on the fly for SQX

I invite you to think seriously about it. If you are interested we can talk about it more technically, there are some conditions to make the gpu computation advantageous, few exchanges of memory between ram and Vram (so basically you should transfer big pool of strategies) and of course a minimum pool size. 

as a starting point of this i propose:

- to make sure, that 100% similar strategies dont go to databank - same net profit, same number of trades, same RDD - because this dont work in 108

- after that make some settings with threshold what i wrote up

- make a decision what next could be done

implemented as proposed in my comment - it filters out same strategies, and it checks for strategies that are within 5% of Net profit, Number of trades and Drawdown. It then keeps the best of these two in the databank.

We can maybe later improve it by checking correlation, etc.

rng - I understand that GPU could be used to compute correlation, however, without using filters strategies can be theoretically saved to databank every few miliseconds, not just once in a while. So I'm not sure it possible to use GPU and keep it synchronized with the actual databank state. But let's discuss it further. I'll contact you directly, I have your email. 

bolo by super ak by bolo pri tych strategiach aj info ze kolko podobnych mala ta ktora strategia. podla mna by to mohlo indikovat istu robustnost. len na zvazenie