SQ Community "Hive Mind" Database for Shared Distributed Deep Learning

Instead of dealing with local datasets, we could allow an option for people to submit strategies to a server storing strategies collectively. Some people can be private and choose not to participate. But it would incentivize users to submit good strategies because the NN would be able to pull data and "learn" from all of them. The NN can then be updated the more people share and theoretically create better strategies for everyone than the current ones they share/find. Regardless, we all win because the larger the dataset grows in size, the better a Neural Network can perform for everyone to make new unique strategies. 

All users would need is an "Upload to Hive Mind" button to click on in order to submit selected strategies. The rest would be managed in the backend. No one would get access to individual strategies hosted on the server. We would just get an updated AI engine on each update with all of the new things it's learned from all the new data it's processed. 

Support for uploading and downloading distributed custom indicators should be apart of this so that users can submit strategies that implement custom indicators. Custom indicators would be the only files available for distributed downloading due to the fact SQ would need them in order to build strategies based on learned custom strategies. If a user does not want their custom indicators uploaded, they cannot post a strategy that implements custom indicators. 

• The need for downloading custom indicators may be solved if a NN was able to learn from custom strategies and use pieces of it to build strategies SQ users can build with the standard set of indicators, however, custom strategies generally rely heavily on the custom indicators that make it up so the data learned from these strategies may be entirely useless if SQ users are not provided the components needed to achieve the performance of that custom strategy.
  

Another neat feature that can be added is distributed computing so all SQ users would be able to contribute computer processing from their own computers in order to help process robustness tests for all of the strategies in the hive. When robustness tests are processed, a results summary can be sent back to and stored on the server and attached to that individual strategy. The Nerual Net would only analyze strategies that have been fully tested and verified. The Neural Network would then weight strategies with good robustness tests higher than ones without and would then be more likely to manipulate blocks in a way to produce more stable strategies for all users. The SQ team could have a standard data set to run tests on for a given instrument (10+ years per dataset, the longer the better). This would solve the issue of people submitting strategies that have been tested at different intervals of time( 2 - 5-year tests). The time frame would then be recomputed from the tick data to match the original time frame of the strategy. If this were added, all that should be needed on the user side is a "Build Hive Mind" button that would prompt the user to dedicate a customizable HD storage/cores/GPU cores to process this data. All the rest should be able to be handled in the backend with comands for retesting and data being supplied in the background.

• Distributed computing can also be used in order to retest all the strategies on tick data on larger datasets in order to ensure the accuracy of results.  The server would use the tick data set to recreate the time period needed based on the metadata provided from the strategy. Running Monte Carlo and statistical tick data retests on hundreds of thousands to millions of strategies would be difficult to maintain with a single server alone. Allowing distributed computing would make achieving this much easier. It would be in everyone's best interest to use the ability to do this on the most accurate data possible so the NN can work better for everyone. The computing power needed to do this would be a lot because tick data is larger than any other resolution, distributed computing solves this issue and will allow us to benefit from learning the robustness of strategies that are verified over longer periods of time. 

• Going a step further, All strategies should be processed using constant risk in order to establish a baseline so data does not have to rely on previous information to calculate risk (I would recommend a $10k balance with $100 per trade risk). This way, you can make a torrent-like structure that you can distribute the data required to run a piece of a test in the background. Two computers would both receive the same data to process and when it is sent, the analysis of both copies of the pieces would be compared in order to be verified (similar to a blockchain structure) where it would then be put back together on the server where it would then be indexed as a strategy to "learn from" in the hive mind. It's a similar concept to crypto mining, but instead of receiving a monetary reward from the computation, we all get an updated AI each new SQ build update. This should be doable after the grid computing interface is completed. 

Think of a day where we can all generate completely random strategies and let the Neural Net, based on collective data we're submitting, to make more of them into much better versions of themselves whereas before, the genetic evolution could discard many strategies completely due to the lack in ability to build off of them. That's essentially the ultimate goal of what this project would accomplish. Take any random strategy and improved it based on commonality patterns in an index of what we have collectively measured to work. The more people who find good strategies from random parameters, the more diverse the Hive Mind will become and thus the more we all will improve as there will be a diverse catalog of patterns to learn from. 

Once stats are fully verified per strategy, the 'learning' should be much more maintainable as it is training from the stored strategy code & stats and making changes based on the characteristics of those blocks rather than training a strategy on an instrument (data intensive) with each iteration.

I would also recommend keeping a maximum on how many strategies the Hive Mind could hold. 1 person can easily submit 1000 strategies over the course of using SQX so I would recommend a size between 1,000,000 and 10,000,000 total strategies to hold to keep a good variety of strategies for the NN to learn from. Any newly submitted strategies would push out strategies at the bottom with the lowest fitness, just like the databank does in SQ. If any retested strategies prove to have much poorer stats when retested, it's fitness should rank low in the Hive Mind. The SQ team can decide if they want to make a filter to ensure bad strategies are excluded from the database, however, the Neural Net should be able to make changes to current strategies based on the commonalities of strategies with higher fitness values. For example, if the NN were to make changes to a strategy based on it having a MACD, it would make decisions based on MACD strategies that had a higher fitness in the database than lower fitness MACD strategies. The more users submit better strategies, the more the database grows which means the NN will make fewer decisions based on lower fitness strategies as they will be pushed towards the bottom of the database until they are deleted completely when the database reaches max capacity.