Generate new strategies instead of too similar ones in gen evo

Now that I´ve got your attention :-)... We all know that even with many islands, genetic evolution will run to a dead end sooner or later - all strategies will look similar, even if using big mutation rates and low crossover rates this cannot be prevented. It´s of course what we want, as we want to find the best candidate from all generations that fits our needs (fitness goal), it´s the whole purpose of genetic evolution, right? I fully agree with that, however, that "perfect strategy" that comes out at the end might just miss this ONE twist some fresh blood would have brought into the whole genetic evolution process. So how could we possibly achieve that? Sure, we can simply start from scratch and run a new genetic evolution, but well, here is another idea that I´ve used with great results elsewhere already:



The initial population is of course generated as usually and as it is implemented right now in SQX, then, on each following sub-generation, too similar members (identical or ALMOST identical equity curve, drawdown, etc. - best detected by a simple correlation analysis of the profit curve) are being replaced by NEW candidates that SQX generates (just like for the initial population) and inserts them in place for the too similar (or exactly identical) strategies in the current generation. Now the next generation does have the slight chance that one of the new randomly generated strategies brings something new/good/better into the genetic pool - and it won´t hurt either, as we´ve just replaced identical or highly identical citizens that wouldn´t have brought us any more benefits for the following generations as they are clones already. The longer the genetic evolution runs, the more candidates in the current generation will become identical/almost identical/clones, and the more of these candidates will be replaced by completely new and fresh random strategies. This can result in that the whole evolution gets a fresh kick and is able to further improve the previously stagnating fitness of the "best candidate" that it found so far. If this isn´t the case, it won´t be worse than it is now: fitness will stagnate and you (or if configured automatically) will cancel the genetic evolution at one point where nothing new happens.


However, most of the time, and from my experience with this method, the genetic evolution basically keeps on running forever, as there will always be fresh blood being inserted. It´s basically the ideal combination of just randomly generating strategies + genetic evolution, as you get the benefits of both method at once: a steady running evolution that improves the whole pool of strategies, and fresh ones that can improve these already existing top candidates and / or will lead to completely new top candidates the longer the genetic evolution runs and the more candidates are being replaced as the generation candidates become too similar. And of course this works perfectly with islands too, as it does nothing more than to simply replace identical "looking" candidates in every generation by random new candidates.


Something to think about and I think this would be a very great addition to SQX, with minimal work needed. And of course, it can be made optional, just in case :-) But I think you guys will be surprised about how powerful this approach is.

Attachments
No attachments
  • Votes +7
  • Project StrategyQuant X
  • Type Feature
  • Status Fixed
  • Priority High

History

g
#1

geektrader

11.01.2019 08:17

Task created

MF
#2

Mark Fric

11.01.2019 09:33

Subject changed from Dracula demands: GIMME FRESH BLOOD! to Generate new strategies instead of too similar ones in gen evo

Priority changed from Normal to High

ok, I changed the title :-) 

Your idea is valid and it is not that difficult to implement. We will make it into the new build.

g
#3

geektrader

11.01.2019 09:40
Argh, that new title is sooooo laaameeee ;-) But OK ;-) haha


Yes, I think you will be surprised how great of an improvement this will make to the genetic evolution. Great to hear you can add it :-)

g
#4

geektrader

11.01.2019 09:40
Voted for this task.
IH
#5

clonex / Ivan Hudec

11.01.2019 09:47
Voted for this task.
N
#6

nathan

11.01.2019 11:53
Voted for this task.
Rr
#7

Partizanas

11.01.2019 15:14
Voted for this task.
BA
#8

Grandmamoses

11.01.2019 18:30
Voted for this task.
rg
#9

Robert Gergelyi

12.01.2019 21:03
Voted for this task.
MF
#10

Mark Fric

16.01.2019 11:23

Status changed from New to Fixed

it is implemented. 


But I tried genetic evolution now and I don't see it generating too many same strategies even without this feature. Do you have some config that generates you a lot of same/similar strategies? If yes, can you attach it here so I can check it?

g
#11

geektrader

16.01.2019 21:08
Hi Mark,


great to hear this. I think we might have misunderstood each other about the use of this feature. I am talking about that the genetic evolution gets stuck in a dead-end after > 300 generations (with 512 population size), meaning it only generates very similar results, as the evolution has converged. This won´t happen anymore to this degree if fresh random strategies are inserted for identical strategies in the current population, hence it will never converge to this degree but keep on running, going "new ways" (finding new "best" strategies). You can basically let it run forever and it will give you a perfect mixture of the random generating/genetic evolution mode of SQX.

MF
#12

Mark Fric

17.01.2019 10:33
I enhanced this functionality a little: https://roadmap.strategyquant.com/tasks/sq4_3977


MG
#13

maegop

03.02.2019 05:27
Voted for this task.

Votes: +7

Drop files to upload

or

choose files

Max size: 5MB

Not allowed: exe, msi, application, reg, php, js, htaccess, htpasswd, gitignore

...
Wait please