[B135] Huge waste of disk space with logs & testfiles

g

#1

geektrader

07.02.2022 14:57

Task created

g

#2

geektrader

07.02.2022 14:58

Voted for this task.

b

#3

bentra

07.02.2022 15:05

Voted for this task.

Bt

#4

beetrader

07.02.2022 15:21

Voted for this task.

l

#5

Loonly

07.02.2022 15:46

Voted for this task.

g

#6

geektrader

08.02.2022 01:31

Description changed:

Having imported 40GB of tick-data, it´s absolutely crazy what amount of LOG files StrategyQuantX generates during this import. This created 130GB log files (with debugging turned off!):

07.02.2022 13:41 131.569.745.797 log_2022_02_07.log 1 File(s) 131.569.745.797 bytes 0 Dir(s) 0 bytes free

So that even my disk even went full while importing.

There is more waste of space in the \StrategyQuantX\internal\testiles directory all the time, as SQX is not deleting temporary files there, even after closing + restarting the application:

Directory of C:\PortablePrograms\StrategyQuantX\internal\testfiles 06.02.2022 02:53 <DIR> . 06.02.2022 02:53 <DIR> .. 06.02.2022 03:30 <DIR> 1 06.02.2022 03:29 <DIR> 2 04.02.2022 04:26 <DIR> 6 06.02.2022 02:53 <DIR> 9 0 File(s) 0 bytes Directory of C:\PortablePrograms\StrategyQuantX\internal\testfiles\1 06.02.2022 03:30 <DIR> . 06.02.2022 03:30 <DIR> .. 03.02.2022 08:54 <DIR> 0 06.02.2022 03:30 <DIR> 2 06.02.2022 03:30 <DIR> 9 0 File(s) 0 bytes Directory of C:\PortablePrograms\StrategyQuantX\internal\testfiles\1\0 03.02.2022 08:54 <DIR> . 03.02.2022 08:54 <DIR> .. 03.02.2022 08:55 1.588.665.460 c-1070338070 1 File(s) 1.588.665.460 bytes Directory of C:\PortablePrograms\StrategyQuantX\internal\testfiles\1\2 06.02.2022 03:30 <DIR> . 06.02.2022 03:30 <DIR> .. 06.02.2022 03:30 206.425.060 c1228218752 06.02.2022 03:30 203.112.371 c1276484229 2 File(s) 409.537.431 bytes Directory of C:\PortablePrograms\StrategyQuantX\internal\testfiles\1\9 06.02.2022 03:30 <DIR> . 06.02.2022 03:30 <DIR> .. 06.02.2022 02:53 4.461.796 c19307747 06.02.2022 03:30 246.183.817 c1932068986 2 File(s) 250.645.613 bytes Directory of C:\PortablePrograms\StrategyQuantX\internal\testfiles\2 06.02.2022 03:29 <DIR> . 06.02.2022 03:29 <DIR> .. 06.02.2022 03:29 <DIR> 0 06.02.2022 02:53 <DIR> 5 0 File(s) 0 bytes Directory of C:\PortablePrograms\StrategyQuantX\internal\testfiles\2\0 06.02.2022 03:29 <DIR> . 06.02.2022 03:29 <DIR> .. 06.02.2022 03:30 203.821.595 c-2089725502 1 File(s) 203.821.595 bytes Directory of C:\PortablePrograms\StrategyQuantX\internal\testfiles\2\5 06.02.2022 02:53 <DIR> . 06.02.2022 02:53 <DIR> .. 06.02.2022 02:53 5.695.115 c-253865061 1 File(s) 5.695.115 bytes Directory of C:\PortablePrograms\StrategyQuantX\internal\testfiles\6 04.02.2022 04:26 <DIR> . 04.02.2022 04:26 <DIR> .. 04.02.2022 04:26 <DIR> 5 0 File(s) 0 bytes Directory of C:\PortablePrograms\StrategyQuantX\internal\testfiles\6\5 04.02.2022 04:26 <DIR> . 04.02.2022 04:26 <DIR> .. 04.02.2022 04:27 1.906.289.122 c653707452 1 File(s) 1.906.289.122 bytes Directory of C:\PortablePrograms\StrategyQuantX\internal\testfiles\9 06.02.2022 02:53 <DIR> . 06.02.2022 02:53 <DIR> .. 06.02.2022 02:53 <DIR> 0 06.02.2022 02:53 <DIR> 5 0 File(s) 0 bytes Directory of C:\PortablePrograms\StrategyQuantX\internal\testfiles\9\0 06.02.2022 02:53 <DIR> . 06.02.2022 02:53 <DIR> .. 06.02.2022 02:53 4.344.183 c-909449818 1 File(s) 4.344.183 bytes Directory of C:\PortablePrograms\StrategyQuantX\internal\testfiles\9\5 06.02.2022 02:53 <DIR> . 06.02.2022 02:53 <DIR> .. 06.02.2022 02:53 4.278.848 c-957715295 1 File(s) 4.278.848 bytes Total Files Listed: 10 File(s) 4.373.277.367 bytes

Please fix the cleanup of these files and stop SQX from creating these huge log files upon tick-data import. This could be done by simply limiting the size of the log files in the settings, so that we can set say "64MB" "max log file size" and older data is overwritten with new one if it exceeds that threshold.

Thank you.

g

#7

geektrader

08.02.2022 03:59

Another update. I´ve taken a look at the huge log file, it´s swamped with 96 million lines totaling at 21GB of these messages when importing tick-data that was exported from MT5:

03:10:32.303 [qtp1177067563-515] INFO c.s.d.data.imports.DataImportEngine - Import history data - recognized file format 'MetaTrader5 Tick Data'. 03:11:15.180 [Nonblocking computeThread - 0 - DataImportFromFile_AUDNZD_ICM_Ticks] INFO c.s.datalib.data.io.DataCsvLoader - [checking_data] Skipping row 10866824, line: 2015.07.23 08:45:32.000 1.11143 1000000.00000000 24, reason: empty String 03:11:15.180 [Nonblocking computeThread - 0 - DataImportFromFile_AUDNZD_ICM_Ticks] INFO c.s.datalib.data.io.DataCsvLoader - [checking_data] Skipping row 10866825, line: 2015.07.23 08:45:33.000 1.11129 200000.00000000 24, reason: empty String 03:11:15.180 [Nonblocking computeThread - 0 - DataImportFromFile_AUDNZD_ICM_Ticks] INFO c.s.datalib.data.io.DataCsvLoader - [checking_data] Skipping row 10866826, line: 2015.07.23 08:45:33.000 200000.00000000 16, reason: empty String 03:11:15.180 [Nonblocking computeThread - 0 - DataImportFromFile_AUDNZD_ICM_Ticks] INFO c.s.datalib.data.io.DataCsvLoader - [checking_data] Skipping row 10866827, line: 2015.07.23 08:45:33.000 1.11142 200000.00000000 24, reason: empty String 03:11:15.180 [Nonblocking computeThread - 0 - DataImportFromFile_AUDNZD_ICM_Ticks] INFO c.s.datalib.data.io.DataCsvLoader - [checking_data] Skipping row 10866828, line: 2015.07.23 08:45:34.000 1.11133 200000.00000000 24, reason: empty String 03:11:15.180 [Nonblocking computeThread - 0 - DataImportFromFile_AUDNZD_ICM_Ticks] INFO c.s.datalib.data.io.DataCsvLoader - [checking_data] Skipping row 10866829, line: 2015.07.23 08:45:34.000 450000.00000000 16, reason: empty String 03:11:15.180 [Nonblocking computeThread - 0 - DataImportFromFile_AUDNZD_ICM_Ticks] INFO c.s.datalib.data.io.DataCsvLoader - [checking_data] Skipping row 10866830, line: 2015.07.23 08:45:35.000 1.11137 200000.00000000 24, reason: empty String 03:11:15.180 [Nonblocking computeThread - 0 - DataImportFromFile_AUDNZD_ICM_Ticks] INFO c.s.datalib.data.io.DataCsvLoader - [checking_data] Skipping row 10866831, line: 2015.07.23 08:45:35.000 1.11134 200000.00000000 24, reason: empty String 03:11:15.180 [Nonblocking computeThread - 0 - DataImportFromFile_AUDNZD_ICM_Ticks] INFO c.s.datalib.data.io.DataCsvLoader - [checking_data] Skipping row 10866832, line: 2015.07.23 08:45:35.000 200000.00000000 16, reason: empty String 03:11:15.180 [Nonblocking computeThread - 0 - DataImportFromFile_AUDNZD_ICM_Ticks] INFO c.s.datalib.data.io.DataCsvLoader - [checking_data] Skipping row 10866833, line: 2015.07.23 08:45:35.000 1.11137 200000.00000000 24, reason: empty String

This most likely relates to this bug that I´ve already reported here: https://roadmap.strategyquant.com/tasks/sq4_8492

E

#8

Emmanuel

09.02.2022 14:42

Voted for this task.

g

#9

geektrader

07.04.2022 21:37

Attachment Untitled.png added

Untitled.png

(131.17 KiB)

Here is how it looks after one night of extensive testing - and SQX never deletes those files, just adds new ones. At least upon restarting, SQX should auto-clean that directory. Otherwise users might not know why their disk clutters up further and further.

a

#10

astral

18.04.2022 20:24

Voted for this task.

k

#11

Karish

16.11.2022 13:29

Noticed this problem a while back,

i created a simple batch solution for this, you can run it before you run SQX, or just add a runner for the batch to perform the batch script that will delete all those logs and then run SQX afterwards.

https://discord.com/channels/426766614734831616/992528947529138278/992529771189780521

JH

#12

Jabezz

17.11.2022 06:36

Voted for this task.

MF

#13

Mark Fric

30.11.2022 12:44

what is in /internal/testfiles are not logs but cached datafeeds that are used during backtests. If you delete them SQ will have to compute them again when you'll restart it and run backtest with the datafeed config.

g

#14

geektrader

30.11.2022 13:06

Thanks for your feedback, this might be the case WITHIN a session, but once SQX is ended and restarted, these files are NOT being reused at all. It creates new ones and leaves the old ones there, even if using the exact same settings, cluttering the directory up until the disk is full. Just try yourself and restart SQX a few times, don´t change any settings, just run the same config. The testfiles directory will grow and grow and grow until the disk is full. So it´s not correct that the files in there are re-used over 2 sessions, only within the same SQX session, if you don´t exit it. And because that is the case, these files should be deleted upon SQX closing.

VH

#15

anttropus

30.11.2022 13:22

Voted for this task.

VH

#16

anttropus

30.11.2022 13:24

But cache file should not have .log extension. It is confusing.

g

#17

geektrader

30.11.2022 14:23

They dont. Its 2 different types of files we are talking about in this bug report.

[B135] Huge waste of disk space with logs & testfiles

History

geektrader

geektrader

bentra

beetrader

Loonly

geektrader

geektrader

Emmanuel

geektrader

astral

Karish

Jabezz

Mark Fric

geektrader

anttropus

anttropus

geektrader