Data storage improvements

Noticed that when data folder is 7-zipped, it goes down to half size.



Compression does not need to be .7z, but even .zip also compresses a lot.

So, there is a room for storing the data in compressed format. Can you pls fix this?


I am referring to the market data stored in disk. Not strategy files.


Data gets written when you update it. Data get read at the beginning of Builder/Retester and kept in memory (which can be uncompressed).


Benefits - less I/O and less disk storage for install and backups.

Attachments
No attachments
  • Votes +1
  • Project StrategyQuant X
  • Type Feature
  • Status Refused
  • Priority Normal

History

t
#1

tnickel

08.12.2020 17:57
Is 7 zipped compression slower or faster?

Half size is only one point, if the speed of saving all strategies takes longer, it make no sense.


MF
#2

Mark Fric

15.12.2020 13:49

Status changed from New to Refused

we did some speed tests, but compressed marked data are much slower to load/save. The data are already compressed, but in our own format for fast reading.


This is especially important for tick data whoch are not cached in memory


Votes: +1

Drop files to upload

or

choose files

Max size: 5MB

Not allowed: exe, msi, application, reg, php, js, htaccess, htpasswd, gitignore

...
Wait please