[Build 107 (RC 7)] Data quality checker/analyser

Feature request for Data Manager module:

- Integrated utility that allows automated tests of source(or imported) history data. Checks for data gaps, continuity, graphical quality representation, allows manual quick fix by editing suspect values.


Thank you.

Attachments
No attachments
  • Votes 0
  • Project StrategyQuant X
  • Type Feature
  • Status Fixed
  • Priority Normal

History

DR
#1

mentaledge

16.11.2017 14:29

Task created

m
#2

mikeyc

16.11.2017 20:47
Hi Darius,


A few questions you would need to answer I suspect in order for the SQ4 team to implement this:


  1. How do you define a gap? Weekend gap would depend on timezone of data. What about holiday periods where data isn't published? 
  2. How do you define quality? For example if you mean to identify spikes which might be erroneous values, how do you know if it is a spike or a real value?
  3. With tick data there is so much data it's hard to graphically represent it.  Where there are spikes in the spread (Bid - Ask), is the Bid incorrect or the Ask or both?
Regards,


Mike
 

DR
#3

mentaledge

17.11.2017 20:12
Unfortunately I'm not the expert in this topic.  Dealing with tick level might be tricky indeed. M1 probably would be enough in majority of cases.


Daniels ideas on this topic are quite interesting though: 


http://mechanicalforex.com/2015/12/is-your-forex-data-full-of-holes-building-a-script-for-the-automatic-evaluation-of-forex-data-quality-using-python.html

http://mechanicalforex.com/2016/01/evaluating-forex-historical-data-quality-a-more-in-depth-view-using-a-python-script.html




MF
#4

Mark Fric

31.07.2018 07:00

Status changed from New to Fixed


Votes: 0

Drop files to upload

or

choose files

Max size: 5MB

Not allowed: exe, msi, application, reg, php, js, htaccess, htpasswd, gitignore

...
Wait please