Crucial - SQX Missing Over-fitting-Avoidance by Stable Regions. -- Analysis by Optimization / k-Nearest Neighbors (kNN).

It would be nice if the optimizer could pick a set where all of the values of the variables are from the middle of a  "fat lump."

"Evaluation and Optimization of Trading Strategies" page 200 attached as an example.

The main point that i am trying to make is that SQX fitting too much, and it wont take any stable regions or neighboring parameters in to equation.

Here are 2 pictures for the illustration.

bentra, You are totally right, I will reconstruct this feature suggestion again and edit it with the propper accurate rules and steps with screenshots etc very soon.

https://youtu.be/LzoewgVWsG4

Thank you so much Mark and the dev-team for paying attention to this task, we would all love to see it in work, hope i explained everything clear about everything,

besides that, we already got like most of the parts of this task already built-into SQX currently, so it should be a simple task to accomplish with the help of some already made functions from the back-end.

Thanks :)

From a book that i published here:

https://discord.com/channels/426766614734831616/786238098153144350/841060475274002432

very interesting part and very related to the topic again:

check the screenshot.

Another thing from the same book, This thing is already mentioned above at the main post of this task,

it is the ability to see and make use of the equity curve results of different parameter sets results,

Aiming with the mouse will give us a popup balloon msg with the exact parameters set,

And SQX could actually read information from those results too, like comparing correlations between our Original parameters set Vs. other parameter sets results etc.

See screenshots:

the first one is the already mentioned at the main post,

the other is from the PDF ebook https://discord.com/channels/426766614734831616/786238098153144350/841060475274002432

karish, I'm trying to understand how this new crosscheck should work.

1. Optimization or crosschec? But from what I understand this is more a different optimization method (where you optimize by one parameter at the time), than a crosscheck.

I roughly understand how it could work as an optimization method - the result will be a strategy where all requested parameters were optimized this way.

2. Charts  - 3d charts are already available. I understand that you want to draw 2d and scatter charts, adding them should be simple.

The problem will be with the individual equity curves, it would be a huge amount of data to save also individual orders for every optimization to be able to draw an equity curve, so I wouldn't do that.

3. FIltering  - If this should be somehow used as crosscheck how do you propose the strategy should be filtered as PASSED/FAILED?

karish, I'm trying to understand how this new crosscheck should work.

1. Optimization or crosschec? But from what I understand this is more a different optimization method (where you optimize by one parameter at the time), than a crosscheck.

I roughly understand how it could work as an optimization method - the result will be a strategy where all requested parameters were optimized this way.

2. Charts  - 3d charts are already available. I understand that you want to draw 2d and scatter charts, adding them should be simple.

The problem will be with the individual equity curves, it would be a huge amount of data to save also individual orders for every optimization to be able to draw an equity curve, so I wouldn't do that.

3. FIltering  - If this should be somehow used as crosscheck how do you propose the strategy should be filtered as PASSED/FAILED?

Hi Mark,

I explained everything in-depth as i could in this task's posts,

it should be easy to understand and implement :)

1. Cross-Check,
    It is more or less the same process as using SPP,  - You got everything correct, Yes.

2. Charts,

    We can dump this idea, no problem, 2D & Scatter chart would do the job perfectly.

3. Filtering,

    I left this question on the table to be frank with you,
    I guess we should try something initially and then when this feature is fully implemented into SQX suggestions will follow,

    As for the initial idea for PASS/FAILED which is actually in other words = How SQX can determine the "Stable Region Area":

    Attached a screenshot that i already attached to this task few posts ago, SQX will sort determine first of all the "Highest Fitness Ranked Point" AKA (Y Axis),

    Then, SQX will look at the surrounding parameters (X Axis) + (Z Axis) to check if the "Highest Fitness Ranked Point" AKA (Y Axis) is not decreased below X% of the original value of the (Y Axis).
    Example: SQX found the highest Y Axis, R/DD=2.0, Then SQX will check if the nearest parameters up to 10 permutations (X Axis) + (Z Axis) got More or Equal amount of R/DD in X%, lets say above 60%.

So to sum it all up,

In the CrossCheck settings before you run this CrossCheck,

User will have those options to setup:
• Ranking Fitness = R/DD,

• "Stable Region Area" = 10. (Which means SQX will look up-to 10 permutations AKA runs on the (X Axis) + (Z Axis)).

• Accept the result if the difference of the "Highest Point" Vs. "Stable Region Area" is > 60%. (Which mean that all the 10 permutations AKA runs on the (X Axis) + (Z Axis) are higher than 60% of the "Highest Point").

Hope you understood, it is really powerful stuff!, cant wait to test it out!,

Thank you for your reply, trying to help however i can with the explanations. :)

It can work like you describe if you optimize by 1 or 2 parameters. But if optimization is automatic you don't choose the exact parameters to optimize and it can optimize many more parameters - possibly tens of them.

Results of which parameters should be considered for stability?

I'm not sure any strategy would pass if we'd do it for all combinations of optimized parameters.

for example that i gave last post i gave an example for a 3D optimization,

which means if a strategy for example got 5 parameters in total,

the CrossCheck will optimize at first the 1st+2nd parameters out of the 5,

Select the highest Fitness result of the 3D chart,

Check if the highest Fitness result got surrounding results X+Z that are >= 60% of the highest fitness result,

If true, than we select those 2 parameters and continue with the next optimization,

Now we keep the 1st parameter out of the 5 as is, and use the 2nd+3rd parameters out of the 5 to do the same process,

again and again, until we cover all the 5 parameters.

EXAMPLE:

[Parameter #1] + [Parameter #2] + [Parameter #3] + [Parameter #4] + [Parameter #5]

[Parameter #1] + [Parameter #2] + [Parameter #3] + [Parameter #4] + [Parameter #5]

[Parameter #1] + [Parameter #2] + [Parameter #3] + [Parameter #4] + [Parameter #5]

[Parameter #1] + [Parameter #2] + [Parameter #3] + [Parameter #4] + [Parameter #5]

we optimize the 1st parameter + the 2nd parameter,

if highest fitness point is surrounded by other parameter combinations of 1st parameter + 2nd parameter with >= than 60%, we apply those 2 parameters to the strategy and continue,

we optimize the 2nd parameter + the 3rd parameter,

if highest fitness point is surrounded by other parameter combinations of 2nd parameter + 3rd parameter with >= than 60%, we apply those 2 parameters to the strategy and continue,

we optimize the 3rd parameter + the 4th parameter,

if highest fitness point is surrounded by other parameter combinations of 3rd parameter + 4th parameter with >= than 60%, we apply those 2 parameters to the strategy and continue,

we optimize the 4th parameter + the 5th parameter,

if highest fitness point is surrounded by other parameter combinations of 4th parameter + 5th parameter with >= than 60%, we apply those 2 parameters to the strategy and continue,

Now we are sure we got a strategy that has stable regions of parameters.

There should be like some kind of a Score value for each re-optimization step i guess,

just like with WFM or so, and than a TOTAL Score at the end or such.

We gotta think it through though, this feature is missing and it will help ALOT!,

Hope i helped..

The CrossCheck will now take the PARAMETER #1 + PARAMETER #2 out of the 5 and Optimize it by user's defined MAX/MIN/STEPS,

For example we got 50 permutation runs out of those 2 parameters out of the 5,

SQX will take the BEST most fit result,

SQX will analyze by looking at the nearest results (UP+DOWN) and check if for example 10 results UP & 10 results DOWN are holding atleast >=60% of the initial "BEST most fit result",
If true, this step will be a PASS,
SQX will now automatically change the original strategy's parameters set to the new parameters set that includes the new 2 parameters values.

STEP #2:

The CrossCheck will now take the PARAMETER #2 + PARAMETER #3 out of the 5 and Optimize it by user's defined MAX/MIN/STEPS,

For example we got 50 permutation runs out of those 2 parameters out of the 5,

SQX will take the BEST most fit result,

SQX will now automatically change the original strategy's parameters set to the new parameters set that includes the new 2 parameters values.

STEP #3:

The CrossCheck will now take the PARAMETER #3 + PARAMETER #4 out of the 5 and Optimize it by user's defined MAX/MIN/STEPS,

For example we got 50 permutation runs out of those 2 parameters out of the 5,

SQX will take the BEST most fit result,

SQX will now automatically change the original strategy's parameters set to the new parameters set that includes the new 2 parameters values.

STEP #4:

The CrossCheck will now take the PARAMETER #4 + PARAMETER #5 out of the 5 and Optimize it by user's defined MAX/MIN/STEPS,

For example we got 50 permutation runs out of those 2 parameters out of the 5,

SQX will take the BEST most fit result,

SQX will now automatically change the original strategy's parameters set to the new parameters set that includes the new 2 parameters values.

STEP #5:

If strategy has passed all those steps, considered the strategy has PASSED this CrossCheck overall.

_
It can be done,
I hope i explained it so everybody can understand it,
This is without using any visual representation of the processes, this is just how its going to work at the background,
Visuals like 2D + 3D charts etc can be added afterwards just to make sure everything is working flawlessly and just for those who want that manual touch to this all.

Thanks Mark and the team, hope to see this feature soon :).

and if FAILS again, try the 3rd best "BEST most fit result",

maybe it could find any stability on those, and get the PASS status and continue to the next steps based on the 2nd+3d parameters etc.. etc.. etc..,

So we wont just dump our strategy if we failed to find stability on the best "BEST most fit result", we try lower "BEST most fit result" and check for stability there.


This thing should be defined by the USER ahead of time, some thing like:

"Retry to find stability by using lower Best Fitness value = 10"

This way SQX will try 10 times per each step to find stability like explained above,
if it fails, just dump this strategy and continue to the next strategy.

But there are few things I'd do differently:

1. why optimizing by 2 parameters in each step? I think it is better to optimize by just one parameter, and then move to another one.

It will be faster (less combinations), and the stability can be recognized better on 2D chart.

2. checking fitness - I'd probably change way the best strategy is determined - instead of trying best fitness, then second best, then third best etc. I'd try to find the stable area on the result chart, and the parameter in the middle of the stable area would be the one used.

And yes if we will choose to use a "2D chart" method rather than the "3D chart" method its going to work the same as i wrote at the previous post:
https://roadmap.strategyquant.com/tasks/sq4_8062#comment-93192

just using only 1 parameter at a time, so yes it will be faster etc., depends on the user's choice i guess. (the choice for 2D or 3D should be included to the user's choices using this cross-check)*.

Refereeing to your 2nd comment:

I guess this way would be more efficient in your point of view, so yeah i would say go-ahead with it, we can always think about better ways to evaluate this in the future with the help of the community etc,
we gotta have a Beta version of this cross-check atleast and then we can really get the feel of it and check what would suits best.

__

Thanks for your reply, looking forward to it :),
good luck!

I am sure we will all benefit from it,

and yeah, 2D sounds good first the initial implementation of this project :), Thanks! excited to see it on the following beta!

You guys rock!!!!!!!!!!!!!!!!!

Thank you so much Mark and the dev team!