<- Back to Tasks

Wrong Memory management

Hi, application grabbing more memory then selected and causing errors.



00:03:05.966 [Blocking computeThread #1 - Strategy 3.16.104674] ERROR MonteCarloCrossCheckMethod - Optimization Strategy 3.16.104 MC 62 failed, error: java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.base/java.lang.Integer.valueOf(Integer.java:1050)
at it.unimi.dsi.fastutil.ints.IntIterator.next(IntIterator.java:43)
at it.unimi.dsi.fastutil.ints.IntIterator.next(IntIterator.java:26)
at com.strategyquant.tradinglib.simulator.impl.MetaTraderSimulatorNetting.updateMAE_MFE(Unknown Source)
at com.strategyquant.tradinglib.simulator.impl.MetaTraderSimulatorNetting.processNewTick(Unknown Source)
at com.strategyquant.tradinglib.simulator.impl.MetaTraderSimulatorNetting.eventNewTick(Unknown Source)
at com.strategyquant.tradinglib.backtest.BacktestDataFeed.processEvent(Unknown Source)
at com.strategyquant.tradinglib.backtest.BacktestDataFeed.start(Unknown Source)
at com.strategyquant.tradinglib.engine.BacktestEngine.runBacktest(Unknown Source)
at com.strategyquant.tradinglib.engine.BacktestEngine.runBacktest(Unknown Source)
at com.strategyquant.plugin.CrossCheck.impl.MonteCarloRetest.MCRetestJob.call(Unknown Source)
at com.strategyquant.plugin.CrossCheck.impl.MonteCarloRetest.MCRetestJob.call(Unknown Source)
at com.strategyquant.tradinglib.simplegrid.SimpleGridEngine.runSingleThreaded(Unknown Source)
at com.strategyquant.tradinglib.simplegrid.SimpleGridEngine.start(Unknown Source)
at com.strategyquant.plugin.CrossCheck.impl.MonteCarloRetest.MonteCarloRetestPlugin.runOnGrid(Unknown Source)
at com.strategyquant.plugin.CrossCheck.impl.MonteCarloRetest.MonteCarloRetestPlugin.runTest(Unknown Source)
at com.strategyquant.tradinglib.backtestrunner.BacktestRunner.runAndSaveCrossChecks(Unknown Source)
at com.strategyquant.tradinglib.backtestrunner.BacktestRunner.execute(Unknown Source)
at com.strategyquant.tradinglib.backtestrunner.BacktestRunner.execute(Unknown Source)
at com.strategyquant.plugin.Task.impl.Retest.RetestJob.call(Unknown Source)
at com.strategyquant.plugin.Task.impl.Retest.RetestJob.call(Unknown Source)
at com.strategyquant.gridlib.compute.performer.MultithreadComputePerformer$1.run(Unknown Source)
at com.strategyquant.gridlib.concurrent.ThreadPool$TaskExecutor.run(Unknown Source)
at java.base/java.lang.Thread.run(Thread.java:834)


00:03:12.273 [Blocking computeThread #1 - Strategy 3.16.104674] ERROR SimpleGridEngine - Error while running task #2
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.base/java.lang.Integer.valueOf(Integer.java:1050) ~[na:na]
at it.unimi.dsi.fastutil.ints.IntIterator.next(IntIterator.java:43) ~[fastutil.jar:na]
at it.unimi.dsi.fastutil.ints.IntIterator.next(IntIterator.java:26) ~[fastutil.jar:na]
at com.strategyquant.tradinglib.simulator.impl.MetaTraderSimulatorNetting.updateMAE_MFE(Unknown Source) ~[SQTradingLib.jar:na]
at com.strategyquant.tradinglib.simulator.impl.MetaTraderSimulatorNetting.processNewTick(Unknown Source) ~[SQTradingLib.jar:na]
at com.strategyquant.tradinglib.simulator.impl.MetaTraderSimulatorNetting.eventNewTick(Unknown Source) ~[SQTradingLib.jar:na]
at com.strategyquant.tradinglib.backtest.BacktestDataFeed.processEvent(Unknown Source) ~[SQTradingLib.jar:na]
at com.strategyquant.tradinglib.backtest.BacktestDataFeed.start(Unknown Source) ~[SQTradingLib.jar:na]
at com.strategyquant.tradinglib.engine.BacktestEngine.runBacktest(Unknown Source) ~[SQTradingLib.jar:na]
at com.strategyquant.tradinglib.engine.BacktestEngine.runBacktest(Unknown Source) ~[SQTradingLib.jar:na]
at com.strategyquant.plugin.CrossCheck.impl.MonteCarloRetest.MCRetestJob.call(Unknown Source) ~[na:na]
at com.strategyquant.plugin.CrossCheck.impl.MonteCarloRetest.MCRetestJob.call(Unknown Source) ~[na:na]
at com.strategyquant.tradinglib.simplegrid.SimpleGridEngine.runSingleThreaded(Unknown Source) ~[SQTradingLib.jar:na]
at com.strategyquant.tradinglib.simplegrid.SimpleGridEngine.start(Unknown Source) ~[SQTradingLib.jar:na]
at com.strategyquant.plugin.CrossCheck.impl.MonteCarloRetest.MonteCarloRetestPlugin.runOnGrid(Unknown Source) ~[na:na]
at com.strategyquant.plugin.CrossCheck.impl.MonteCarloRetest.MonteCarloRetestPlugin.runTest(Unknown Source) ~[na:na]
at com.strategyquant.tradinglib.backtestrunner.BacktestRunner.runAndSaveCrossChecks(Unknown Source) ~[SQTradingLib.jar:na]
at com.strategyquant.tradinglib.backtestrunner.BacktestRunner.execute(Unknown Source) ~[SQTradingLib.jar:na]
at com.strategyquant.tradinglib.backtestrunner.BacktestRunner.execute(Unknown Source) ~[SQTradingLib.jar:na]
at com.strategyquant.plugin.Task.impl.Retest.RetestJob.call(Unknown Source) ~[na:na]
at com.strategyquant.plugin.Task.impl.Retest.RetestJob.call(Unknown Source) ~[na:na]
at com.strategyquant.gridlib.compute.performer.MultithreadComputePerformer$1.run(Unknown Source) ~[SQGridLib2.jar:na]
at com.strategyquant.gridlib.concurrent.ThreadPool$TaskExecutor.run(Unknown Source) ~[SQGridLib2.jar:na]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
00:04:09.929 [Blocking computeThread #2 - Strategy 1.14.159675] ERROR MonteCarloCrossCheckMethod - Optimization Strategy 1.14.159 MC 23 failed, error: java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.base/java.lang.Integer.valueOf(Integer.java:1050)
at it.unimi.dsi.fastutil.ints.IntIterator.next(IntIterator.java:43)
at it.unimi.dsi.fastutil.ints.IntIterator.next(IntIterator.java:26)
at com.strategyquant.tradinglib.simulator.impl.MetaTraderSimulatorNetting.updateMAE_MFE(Unknown Source)
at com.strategyquant.tradinglib.simulator.impl.MetaTraderSimulatorNetting.processNewTick(Unknown Source)
at com.strategyquant.tradinglib.simulator.impl.MetaTraderSimulatorNetting.eventNewTick(Unknown Source)
at com.strategyquant.tradinglib.backtest.BacktestDataFeed.processEvent(Unknown Source)
at com.strategyquant.tradinglib.backtest.BacktestDataFeed.start(Unknown Source)
at com.strategyquant.tradinglib.engine.BacktestEngine.runBacktest(Unknown Source)
at com.strategyquant.tradinglib.engine.BacktestEngine.runBacktest(Unknown Source)
at com.strategyquant.plugin.CrossCheck.impl.MonteCarloRetest.MCRetestJob.call(Unknown Source)
at com.strategyquant.plugin.CrossCheck.impl.MonteCarloRetest.MCRetestJob.call(Unknown Source)
at com.strategyquant.tradinglib.simplegrid.SimpleGridEngine.runSingleThreaded(Unknown Source)
at com.strategyquant.tradinglib.simplegrid.SimpleGridEngine.start(Unknown Source)
at com.strategyquant.plugin.CrossCheck.impl.MonteCarloRetest.MonteCarloRetestPlugin.runOnGrid(Unknown Source)
at com.strategyquant.plugin.CrossCheck.impl.MonteCarloRetest.MonteCarloRetestPlugin.runTest(Unknown Source)
at com.strategyquant.tradinglib.backtestrunner.BacktestRunner.runAndSaveCrossChecks(Unknown Source)
at com.strategyquant.tradinglib.backtestrunner.BacktestRunner.execute(Unknown Source)
at com.strategyquant.tradinglib.backtestrunner.BacktestRunner.execute(Unknown Source)
at com.strategyquant.plugin.Task.impl.Retest.RetestJob.call(Unknown Source)
at com.strategyquant.plugin.Task.impl.Retest.RetestJob.call(Unknown Source)
at com.strategyquant.gridlib.compute.performer.MultithreadComputePerformer$1.run(Unknown Source)
at com.strategyquant.gridlib.concurrent.ThreadPool$TaskExecutor.run(Unknown Source)
at java.base/java.lang.Thread.run(Thread.java:834)


00:04:09.936 [Blocking computeThread #2 - Strategy 1.14.159675] ERROR SimpleGridEngine - Error while running task #3
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.base/java.lang.Integer.valueOf(Integer.java:1050) ~[na:na]
at it.unimi.dsi.fastutil.ints.IntIterator.next(IntIterator.java:43) ~[fastutil.jar:na]
at it.unimi.dsi.fastutil.ints.IntIterator.next(IntIterator.java:26) ~[fastutil.jar:na]
at com.strategyquant.tradinglib.simulator.impl.MetaTraderSimulatorNetting.updateMAE_MFE(Unknown Source) ~[SQTradingLib.jar:na]
at com.strategyquant.tradinglib.simulator.impl.MetaTraderSimulatorNetting.processNewTick(Unknown Source) ~[SQTradingLib.jar:na]
at com.strategyquant.tradinglib.simulator.impl.MetaTraderSimulatorNetting.eventNewTick(Unknown Source) ~[SQTradingLib.jar:na]
at com.strategyquant.tradinglib.backtest.BacktestDataFeed.processEvent(Unknown Source) ~[SQTradingLib.jar:na]
at com.strategyquant.tradinglib.backtest.BacktestDataFeed.start(Unknown Source) ~[SQTradingLib.jar:na]
at com.strategyquant.tradinglib.engine.BacktestEngine.runBacktest(Unknown Source) ~[SQTradingLib.jar:na]
at com.strategyquant.tradinglib.engine.BacktestEngine.runBacktest(Unknown Source) ~[SQTradingLib.jar:na]
at com.strategyquant.plugin.CrossCheck.impl.MonteCarloRetest.MCRetestJob.call(Unknown Source) ~[na:na]
at com.strategyquant.plugin.CrossCheck.impl.MonteCarloRetest.MCRetestJob.call(Unknown Source) ~[na:na]
at com.strategyquant.tradinglib.simplegrid.SimpleGridEngine.runSingleThreaded(Unknown Source) ~[SQTradingLib.jar:na]
at com.strategyquant.tradinglib.simplegrid.SimpleGridEngine.start(Unknown Source) ~[SQTradingLib.jar:na]
at com.strategyquant.plugin.CrossCheck.impl.MonteCarloRetest.MonteCarloRetestPlugin.runOnGrid(Unknown Source) ~[na:na]
at com.strategyquant.plugin.CrossCheck.impl.MonteCarloRetest.MonteCarloRetestPlugin.runTest(Unknown Source) ~[na:na]
at com.strategyquant.tradinglib.backtestrunner.BacktestRunner.runAndSaveCrossChecks(Unknown Source) ~[SQTradingLib.jar:na]
at com.strategyquant.tradinglib.backtestrunner.BacktestRunner.execute(Unknown Source) ~[SQTradingLib.jar:na]
at com.strategyquant.tradinglib.backtestrunner.BacktestRunner.execute(Unknown Source) ~[SQTradingLib.jar:na]
at com.strategyquant.plugin.Task.impl.Retest.RetestJob.call(Unknown Source) ~[na:na]
at com.strategyquant.plugin.Task.impl.Retest.RetestJob.call(Unknown Source) ~[na:na]
at com.strategyquant.gridlib.compute.performer.MultithreadComputePerformer$1.run(Unknown Source) ~[SQGridLib2.jar:na]
at com.strategyquant.gridlib.concurrent.ThreadPool$TaskExecutor.run(Unknown Source) ~[SQGridLib2.jar:na]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
00:04:33.845 [Blocking computeThread #1 - Strategy 3.16.104674] ERROR MonteCarloCrossCheckMethod - Optimization Strategy 3.16.104 MC 63 failed, error: java.lang.OutOfMemoryError: GC overhead limit exceeded


00:04:33.846 [Blocking computeThread #1 - Strategy 3.16.104674] ERROR SimpleGridEngine - Error while running task #3
java.lang.OutOfMemoryError: GC overhead limit exceeded
00:05:18.959 [Blocking computeThread #0 - Strategy 4.11.108676] ERROR MonteCarloCrossCheckMethod - Optimization Strategy 4.11.108 MC 0 failed, error: java.lang.OutOfMemoryError: GC overhead limit exceeded


00:05:18.968 [Blocking computeThread #0 - Strategy 4.11.108676] ERROR SimpleGridEngine - Error while running task #0
java.lang.OutOfMemoryError: GC overhead limit exceeded
00:05:42.588 [Blocking computeThread #1 - Strategy 3.16.104674] ERROR MonteCarloCrossCheckMethod - Optimization Strategy 3.16.104 MC 64 failed, error: java.lang.OutOfMemoryError: GC overhead limit exceeded


00:05:42.592 [Blocking computeThread #1 - Strategy 3.16.104674] ERROR SimpleGridEngine - Error while running task #4
java.lang.OutOfMemoryError: GC overhead limit exceeded
00:06:21.377 [Thread-182] INFO  c.s.t.project.ProgressEngine - Probíhá synchronizace databanky 'TMP'...
00:06:36.117 [Blocking computeThread #2 - Strategy 1.14.159675] ERROR MonteCarloCrossCheckMethod - Optimization Strategy 1.14.159 MC 24 failed, error: java.lang.OutOfMemoryError: GC overhead limit exceeded


00:06:36.128 [Blocking computeThread #2 - Strategy 1.14.159675] ERROR SimpleGridEngine - Error while running task #4
java.lang.OutOfMemoryError: GC overhead limit exceeded
00:07:12.368 [Blocking computeThread #1 - Strategy 3.16.104674] ERROR MonteCarloCrossCheckMethod - Optimization Strategy 3.16.104 MC 65 failed, error: java.lang.OutOfMemoryError: GC overhead limit exceeded


00:07:12.368 [Blocking computeThread #1 - Strategy 3.16.104674] ERROR SimpleGridEngine - Error while running task #0
java.lang.OutOfMemoryError: GC overhead limit exceeded
00:07:16.631 [Thread-182] INFO  c.s.t.project.ProgressEngine - Databank 'Retester - TMP' synchronized to files - 1 records updated in 51.31 s.
00:07:53.076 [Blocking computeThread #0 - Strategy 4.11.108676] ERROR MonteCarloCrossCheckMethod - Optimization Strategy 4.11.108 MC 1 failed, error: java.lang.OutOfMemoryError: GC overhead limit exceeded


00:07:53.080 [Blocking computeThread #0 - Strategy 4.11.108676] ERROR SimpleGridEngine - Error while running task #1
java.lang.OutOfMemoryError: GC overhead limit exceeded
00:08:15.465 [Blocking computeThread #0 - Strategy 4.11.108676] ERROR MonteCarloCrossCheckMethod - Optimization Strategy 4.11.108 MC 2 failed, error: java.lang.OutOfMemoryError: GC overhead limit exceeded


00:08:15.466 [Blocking computeThread #0 - Strategy 4.11.108676] ERROR SimpleGridEngine - Error while running task #2
java.lang.OutOfMemoryError: GC overhead limit exceeded
00:08:36.944 [Blocking computeThread #1 - Strategy 3.16.104674] ERROR MonteCarloCrossCheckMethod - Optimization Strategy 3.16.104 MC 66 failed, error: java.lang.OutOfMemoryError: GC overhead limit exceeded


00:08:36.944 [Blocking computeThread #1 - Strategy 3.16.104674] ERROR SimpleGridEngine - Error while running task #1
java.lang.OutOfMemoryError: GC overhead limit exceeded
00:09:31.822 [Blocking computeThread #2 - Strategy 1.14.159675] ERROR MonteCarloCrossCheckMethod - Optimization Strategy 1.14.159 MC 25 failed, error: java.lang.OutOfMemoryError: GC overhead limit exceeded


00:09:31.834 [Blocking computeThread #2 - Strategy 1.14.159675] ERROR SimpleGridEngine - Error while running task #0
java.lang.OutOfMemoryError: GC overhead limit exceeded
00:09:59.750 [Blocking computeThread #1 - Strategy 3.16.104674] ERROR MonteCarloCrossCheckMethod - Optimization Strategy 3.16.104 MC 67 failed, error: java.lang.OutOfMemoryError: GC overhead limit exceeded


00:09:59.762 [Blocking computeThread #1 - Strategy 3.16.104674] ERROR SimpleGridEngine - Error while running task #2
java.lang.OutOfMemoryError: GC overhead limit exceeded
00:10:50.208 [Blocking computeThread #1 - Strategy 3.16.104674] ERROR MonteCarloCrossCheckMethod - Optimization Strategy 3.16.104 MC 68 failed, error: java.lang.OutOfMemoryError: GC overhead limit exceeded

Attachments
No attachments
  • Votes +1
  • Project StrategyQuant X
  • Type Bug
  • Status Refused
  • Priority Normal

History

ZD
#1

ZdenekDrizga

24.07.2020 13:04

Task created

MF
#2

Mark Fric

28.07.2020 09:16

Status changed from New to Refused

no, the messages mean that Java used all the available memory. How much memory do you have configued?


You probably need to increase it or decrease number of optimization steps.

ZD
#3

ZdenekDrizga

28.07.2020 12:00
I have set a limit for memory to 7 GB, application is taking more then 7,5 GB. I have expected thetr your app control using of memory itselfs. Would be posible to have some regural warning or stop process instead of crash?
MF
#4

Mark Fric

28.07.2020 12:26
I'm not sure how much memory you have on your PC, 7 GB for SQ is relatively small if you store a lot of strategies in the databank.


There should be no problem setting it to use 16-32 GB, if you hav eenough RAM.  SQ will use only the needed amount.


> Would be posible to have some regural warning or stop process instead of crash?


we have a check there, but it is not 100%, because it is difficult to detect. You should configure SQ to use less strategies or optimizations if it is causing this kind of error.

ZD
#5

ZdenekDrizga

28.07.2020 21:17
Hi,


agree that number of strategies can impact the usage of memory, some warnings or recommendations, or test should be good in the new version.

Anyway, an application should be able to correctly handle such exceptions instead of the crash at all or block computer, especially if there is possible to set a limit.


OutOfMemoryError is a quite common basic knowledge of its cause and the solution is largely unknown among junior developers.  Java Performance The Definitive Guide By Scott Oaks or the Java Performance by Binu John. They are an excellent resource for senior Java developers and also teaches you tools and process to deal with an error like java.lang.OutOfMemoryError.There is a lot of solutions or guidance, like parameters or serialization or another way of memory leaks optimization.


at  least some tips, which can help:

java - Xmx - parameter for JVM


https://betsol.com/java-memory-management-for-java-virtual-machine-jvm/



MF
#6

Mark Fric

29.07.2020 07:53
thank you for teaching me about this, but it is not as simple as you might think :-)

I have some experience with Java, its memory management, garbage collectors and memory leaks.


You can configure SQ to use more memory in the Configuration (icon on top right corner), it does it exactly by setting -Xmx on startup.

ZD
#7

ZdenekDrizga

29.07.2020 09:47
Voted for this task.
ZD
#8

ZdenekDrizga

29.07.2020 09:55
Yes, I agree, 

"it is not as simple as you might think :-)"

especially to push developers to do the right things wich has no priority for them or can be explained or excused with some tech words. I am not trying to dishonest you, or be sarcastic.

I am trying to give you my experience and point of view, from a person who starts to be a manager after 20years of a career as a developer.


I wish you good luck.



MF
#9

Mark Fric

29.07.2020 10:32
this has very high priority for us, but it doesn't have a simple solution. 


We did what we could on the programming side, but we cannot prevent users from using the program in a way that it uses all available memory.

There are so many things that can cause it - too many strategies in databank, big Monte carlo simulations, etc. We don't want to add restrictions everywhere, and once the OutOfMemory error happens it is already too late.



It is up to user to configure his memory right if he experiences such problem. It can be done directly in SQ configuration.

Votes: +1

Drop files to upload

or

choose files

Max size: 5MB

Not allowed: exe, msi, application, reg, php, js, htaccess, htpasswd, gitignore

...
Wait please