<- Back to Tasks

Custom Projects Get Stuck

This happens often, with no real rhym or reason. Custom projects can run several successful cycles, then get stuck at some random point. In the example below this one had run around 5 cycles yesterday, then at 02:26am got stuck on a WF crosscheck and never got out of the loop with processor activity at around 6%. Custom projects cannot be left alone and must be watched constantly for errors and bugs.



I am curious why things like this and the other 6 bugs I posted to do with custom projects did not crop up during basic testing. They are all mainstream bugs as far as I can tell and they surely would have arisen with the most basic functionality testing.. Can it be something local?

Attachments
custom project gets stuck.jpg
custom project gets stuck.jpg
(1.13 MiB)
  • Votes +1
  • Project StrategyQuant X
  • Type Bug
  • Status Fixed
  • Priority Normal

History

N
#1

nathan

17.01.2019 08:22

Task created

N
#2

nathan

17.01.2019 09:17
Here is a video: https://www.dropbox.com/s/ifts9sb926xnkvk/Stuck%20on%20WF%20Cross%20Checks.mov?dl=0


All data prepared at 08:09:51 and first WF test completes 3 seconds later at 08:09:51...this is normal, takes about 3 seconds per WF test, processors at pretty much 100% the whole time.


In the video you can see it gets stuck and never moves on. Processor activity drops to around 20% the whole time. It will remain in this state forever. 


Sometimes it can complete a few more strategies thenget stuck. Sometimes a few project cycles then get stuck. But it will always get stuck somewhere in the project tree on some WF test at some point.


Rebooting computer and/or SQX doesn't solve anything.


N
#3

nathan

17.01.2019 15:29

Attachment log_2019_01_17.log added

Here is the log from today. I have tried to get it running multiple times without success...gets through a handful of strategies than gets stuck...sometimes it will many a batch of 5 or 6 fairly quickly, then get stuck for around 5 or 10 minutes with processor idle, then manage another handful (maybe) but eventually it gets stuck forever...



N
#4

nathan

17.01.2019 15:51

Attachment Error.jpg added

Decided to try clearing out the results databank and starting the project from scratch...which threw the attached error.


It feels like the more I use the application, the more it damages itself internally...sort of like when you get a serious disk error, and if you keep using the disk it rapidly propagates across the disk creating more errors...maybe its that ?! :)

N
#5

nathan

17.01.2019 17:43

Attachment Ranking.jpg added

I reversed the only changes I made to the project in the last 24 hours.


Namely, I was building the strategies with a weighted fitness, and forgot to also put this weighted fitness ranking in each of the other project stages. I corrected this last night so each stage had the weighted fitness as a ranking. After that, things got really bad with the freezing.


I undid this one by one, starting with the crosscheck retest real tick, real spread, setting it back to return/drawdown ratio. And that fixed it. For every stage of the project, walk forward included. Even the builder stage runs significantly faster. All other stages though remain on the corrected weighted fitness ranking.


Going into the real tick, real spread settings, it now shows no selection at all for the fitness? See screen grab. But the project itself is now running smoothly.


This real tick, real spread retest crosscheck stage is one of the ones I have most issues with when it comes to the entanglement bug (https://roadmap.strategyquant.com/tasks/sq4_3972). For example, in addition to this bug, this stage absolutely refuses to retain the commission setting in data, no matter how I save the config, or project. It is as if this stage itself is corrupt, if such a thing is possible. I have no clue.

N
#6

nathan

17.01.2019 18:07
Spoke too soon. Its in a loop again. It appears that changing any setting on the part of the project in which it is stuck, then restating it from that point does something and moves it along, usually for at least 1 project cycle until it gets stuck on something else again
N
#7

nathan

17.01.2019 18:33
Thought I would try recreating the project and see if that helped. Nope.


Now, it starts but doesn't really start, and cannot be stopped, but can be started, but cant because, apparently it is already started. But it inst. Nothing ever happens. Bizarre. See video. 


https://www.dropbox.com/s/3nr0m5nxp8nwkh9/Custom%20Projected%20has%20started%20but%20hasnt%20and%20cannot%20be%20stopped.mov?dl=0

TT
#8

Tamas

18.01.2019 08:50
Please attach the project config.
N
#9

nathan

19.01.2019 10:45

Attachment project.cfx added

Here you.


I 'cloned' the project, which appeared tohelp for a while...it ran for about 21 hours (8 cycles) before getting stuck again. Now its getting stuck randomly again.


Note, if you clone it, you need to reboot SQX before it will work at all. You cannot clone projectsthen run the new clone, you first need to reboot SQX. This applies to any custom project.


Anyway, the getting stuck bug is worse as it means custom projects cant be left to run on their own.

N
#10

nathan

22.01.2019 07:59

Attachment Stuck.jpg added

Attachment log_2019_01_22.log added

Attachment log_2019_01_21.log added

Attachment log_2019_01_20.log added

Attachment log_2019_01_19.log added

Stuck again. Attached the logs (it was in debug mode).


Gets stuck on 21 jan around 21:20

N
#11

nathan

23.01.2019 10:44
Voted for this task.
MF
#12

Mark Fric

12.04.2019 11:30

Status changed from New to Fixed

I set this as fixed, we have new task that deal with it.


It is still not perfect, but we found the most possible reason - maxing memory usage in Java.

Votes: +1

Drop files to upload

or

choose files

Max size: 5MB

Not allowed: exe, msi, application, reg, php, js, htaccess, htpasswd, gitignore

...
Wait please