Ability to clean data already in data manager

Very important thing is to use data (especially tick level data) free of abnormal spikes and gaps that distorts any indicator and thus strategy. Even commonly used dukascopy data has these and for now data manager in SQX can spot these things but we cannot do anything about it here except exporting it, do some cleaning manually/use external scripts and import it again which is not optimal at all. Maybe we could have some simple tools already on data manager when we inspect quality of data? Simplest one would be to have ability to sort price lines by % change of previous line and by this sort all from biggest and then we could select with lines we want to delete. But I think there are more advanced users in data cleaning field from me and could suggest some good ways to handle this easy and effectively.



Two things to have in mind:
1. Besides of tools/ways we could have to clean data and sort/inspect price lines it is super important to remember there are also weekend gaps that are natural thing - this should have some marks its weekend gap or other kind of protection from messing with this kind.
2. If we modify/clean data from for example dukascopy downloaded by build in data manager downloader, updating that data should NOT overwrite our old cleaned parts with downloaded ones. Or at least it should ask about it.

Attachments
  • Votes +7
  • Project StrategyQuant X
  • Type Feature
  • Status New
  • Priority Normal

History

a
#1

astral

20.10.2022 12:11

Task created

JJ
#2

jjsb41

20.10.2022 12:26
Voted for this task.
b
#3

bentra

20.10.2022 12:56
Voted for this task.
CG
#4

Chris G

20.10.2022 14:01
Voted for this task.
E
#5

Emmanuel

20.10.2022 15:10
Voted for this task.
KB
#6

kbtech

20.10.2022 18:22
Voted for this task.
k
#7

Karish

20.10.2022 20:49
Voted for this task.
k
#8

Karish

20.10.2022 20:53

Would be awesome to see more statistics on the data 4sure,


We already got a visual of it in terms of green\red bars,

But some conclusions needed.



Moreover, an automated feature of data "normalisation" which will fix all the holes etc would be great.

TV
#9

Tomas Vanek

26.10.2022 09:54

Attachment image-0.png added

Attachment image-1.png added

image-0.png
(171.05 KiB)
image-1.png
(60.43 KiB)
it is already possible to do it the cleaning manually. See attached screenshots
a
#10

astral

26.10.2022 10:06
That is great Tomas! But we still need something to make this process easier than jumping from table of thousands of data lines to graph and back. Maybe simple another column on data view that could simply show differences between high/low or open/close to previous line (maybe in percentages or something)? Then sorting by that column would easly show us gaps and anomalies. Or making it semi-automated with clickable option to "delete all lines with difference bigger than X%"? This needs little more thinking to exclude weekend gaps from that batch deleting command.
VH
#11

anttropus

04.11.2022 22:09
Voted for this task.

Votes: +7

Drop files to upload

or

choose files

Max size: 5MB

Not allowed: exe, msi, application, reg, php, js, htaccess, htpasswd, gitignore

...
Wait please