Improve genetic evolution effectiveness

I tested EURUSD with small population setting too - 7. Small population (7) has more unstable fitness curve then bigger (100) during evolution and stagnation must exist in smart population evolution with bigger probability.

I stopped test after 3087 generations - I need time for building strategy and not to test this bug again and again. See attached files.

Yes.

I am thinking to stop betatesting from my side till this problem will be solved, because due this is SQX time "uneconomical" - produce smaller amount of strategy into databank per time or produce more similar strategies - waste my computer time and for my will be better my computers time use for work with SQ 3.8.2 then for SQX with this problem.

I still have to look at genetic evolution, it seems it doesn't work optimally right now. It produces strategies slower than Random generation.

Part of this unefficiency is/was due problem with stagnation in my opinion. For example if I set stagnation to 3 and max number of generation to 100. Genetic calculation stop and reset and start with new calculation after 6 - 15 generation if stagnation settins is functional (by SQ 3). If strategies is upgraded by genetics very well on IS, stagnation appear after 15 -50 (80) generations. But if stagnation setting is ignored (as in this reported bug) genetics calculate to  reach 100 generation for each initial population without dependency if initial population is good or bad and this take lot of time and increase (for this setting) genetics unefficiency.

Other problems can be due internal genetic unefficiency calculation as you wrote.

1 observation: In my opinion can be usage of islands in some cases/setting contra productive. Because if I use for example 10 islands genetic calculation start if initial population reach probably 60 - 70 % of target amount (when 7 islands from 10 have of own initial amount filled). Some islands are by time more advanced then other by time calculationa start early then for aother islands and can have better IS fitness (by statistic) but their OOS fitness can be poor, but due migration between islands can spread (due better IS fitness) to all islands and all islands genetic calculation can be uneffectively by bad OOS fitness of original set on for example 1 islands with very good IS fitness but bad OOS fitness.

Reason for this is in my opinion:

1. to use minimal migration (set migration after bigger of generation calculation (20 and more))

2. set amount to migrate <5 %

3. start all genetic calculation on all islands in the same time after total initial populations for all islands is created (not for some islands early then for other as is in build 107 now)!

I forgott to add for point 3 this:

My notice is for first initial population and start first genetic calculation for all islands.

But this problem continue as calculation for some islands end earlier then for other (if stagnation setting is functional) and after probably start probably the other initial population creation for early ended genetic calculation on some islands - for this islands more advanced IS fitness form longer genetic calculation islands can overreproduce this new islands calculation due migration.

I suggest to restart other new initial population for all islands after last genetic calculation ends on all last islands from previous all islands calculation set (IP + islands genetics calculations). And do not start new set of IP+genetic calc. befor the older set ends.

Genetic evolution should be generally made more efficient.

SQX build 108

OK, added info graphs in builder are very usable for my. Stagnation is functional now - I had to increase stagnation setiing for longer genetic calc. - is better for observing how it is calculating. :)

As I see genetic calc. start when first initial population "pocket" for 1st island is full, after start for 2nd etc.. This mean this earlier start population have UNFAIR time/generation advantage in fitness then later start population on other islands in the same set of islands+initial population+genetic on this island.(Printscreens are for first set of calculation after Builder was started) This is bad for my opinion. All islands gen. calc. must start at the same time together in my opinion without different time unfair advantage for different islands.

The islands wit earlier start with mores advanced fitness are spreaded with higher probability to other island by migration. See the discussion above for my point 3.

The 1st printscreen is in moment when 1st island start calculation. 2nd is moment later when 2nd start calculation too. 3rd island is still waiting for initial population filling and filtering to 30 (I had setting for 30 population on 3 islands). Builder config included too.

Probably better is to have genetic calc. on all island in the same generation inn the same time, but I am not specialist programmer in this and i have no idea is this is some program code problem - some backtest during gen. calc. can be quickier then for other islands (lower number of trades, lower strategy code complexity for population on some island then on other etc.) and will be waiting for other island with slower strategies backtest during genetic calc.

SQX build 108

Thanks new info window about genetic in build 108 i see more clearly situation with usage 2 and more islands.

I had set stagnation to 9, max generation to 150 and small population size 7. For removing migration affect set migration after 100 generation and migration rate to minimum 1 %.

If I start the same setting but for 1 island only due stagnation  (9) genetic ends after 9 - c. 20 generation as I observed.

After I set 2 islands with the same settings and see repeatedly how earlier starting island with genetic is waiting in stagnation for 30 -60 generation then later starting siland genetic reach stagnation for 9 generation. After all islands are reset in the same time and new set package calculations start again from beginning. It is bad Because genetic on some islands wastes time for calculation in stagnation more then 9.

If I set 10 islands some island reach over 100 generation in stagnation (not max 9 as is in setting) due this islands do waste calculation in stagnation and are waiting for stagnation of last started island evolution. Due this is more Island calculation is less effective then use 1 island where is from build 108 stagnation normal functional

Generally it is opposite then we need - islands genetics start in different time  with unfair time advantage for earlier start islands, see my notes fro my point 3 above, and all islands end in the same time after the latest started island evolution end - other islands are waiting and calculating in long stagnation for this last island. Yes this frame ideas setting (time different start, the same time end) can be use for mathematical modeling of biological system - included reproducing rate - due this and at the end to see how are in the same artifical "animals" developed on each island. But we use this generally for different purpose and need to do this in opposite (start in the same time, end in different time). Because for us are for example strategies with for example 10 trades per 15 years not usable (but have higher "reproduce rat - quickier backtest and genetic calculation or quickier initial population creation) the normal strategies with for example 400 trades per 15 years for M30 timeframe. We must set the same condition for all kind of strategy befor stagnation - set start genetic calculation in the same time for all islands and end genetic calculation in different time by local stagnation for each island separately. Yeas some island can stop calculation after 20 generation, some after 9, some after 60 generation etc. OK We can use  free CPU power for increase rate calculation for the rest islands befor stagnation or prepare new set package for new initial population (it better in my opinion to use free CPU time to increase rate for rest islands befor stagnation).

I started this repeatedly c. 10x and observed behaviotr of this setting continuously.

In printscreen is config of builder  for 2 islands, genetic setting printscreen and printscreen near 1 island is calculating(waiting) in long stangnation and other is befor stagnation reaching (9) and 2nd printscreen is after reseting after reaching 9 stagnation for  island No. 0 (the numbers 6 in empty initial populations is bug reported in bug report sq4_2451).

I had to do printsreens quickly, due calculation in this setting is relatively quick.

The fitness graph is for No. 0 not 1 - bad name of this graph - it was reported in sq4_2445 bug report .

build 108

- if i set only 1 island my PC could produce 60000 str per hour

- if i set 4 island i could go up to 150000 strs per hour

so by my findings is better to use more islands, but i dont want to use migration betwwen them and want to generate strs from every island independently. So i set migration to 100th generation, so it will never get to it

but i found, that the function "restart evolution" doesnt work independently for each island - but the restart working for all islands together

because in my PC, if i have more than 1 island, the strs per hour could be doubled

"but i found, that the function "restart evolution" doesnt work independently for each island - but the restart working for all islands together"

Yes, this is what I described above in my long comments - especially in red point 3 too and it is wrong.

We need start evolution on all islands in the same time and end evolution for each island independently by calculating his own stagnation. Now SQX genetic on more islands works to the contrary (in opposite way) and this degrades efficiency of islands genetic calculation.

Also, restart after finish and restart because of stagnation are now applied to every island independently, so islands don't wait for each other.

There is surely still some room for improvement, but it is a big step forward.

We made a lot of improvements, but there is still work to be done and I'll focus on this in the next build.

Comparison and Analysis of Different Mutation Strategies to improve the Performance of Genetic Algorithm 

I investigated it and found the bug - it was incorrectly updating previous fitness and so it didn't detect stagnation properly.

This is fixed now and indeed restart because of stagnation happens much more often.