ERROR Download Data From Coinbase

I want to download data for BTCUSD, from Coinbase, but it ends at date 2018-08-10. please check.
Attachments
  • Votes +2
  • Project StrategyQuant X
  • Type Bug
  • Status Fixed
  • Priority Normal

History

AA
#1

Alex

06.06.2022 08:39

Task created

E
#2

Emmanuel

07.06.2022 14:05
Voted for this task.
AA
#3

Alex

18.06.2022 11:08
Voted for this task.
g
#4

Lee Guan Chuan

12.08.2022 15:31

Status changed from New to Fixed

AA
#5

Alex

26.09.2022 14:22

Attachment log_2022_09_26.log added

log_2022_09_26.log
(139.04 KiB)
Hi,

I checked using SQX 136 Dev4, it still ends with error.


AA
#6

Alex

26.09.2022 14:31

Attachment log_2022_09_26.log added

log_2022_09_26.log
(152.12 KiB)
And I clicked again on download, and stopped after say 1 minutes, then appeared that already data for BTC and ETH downloaded until 2019 and 2017.
AA
#7

Alex

26.09.2022 14:45

Attachment Screenshot 2022-09-26 112753.png added

Attachment log_2022_09_26.log added

log_2022_09_26.log
(169.91 KiB)
Screenshot 2022-09-26 112753.png
(6.71 KiB)
Again another error
TT
#8

Tamas

26.09.2022 16:51

Status changed from Fixed to Waiting for information

Hello Alex, please check your internet connection.


From log file => Server returned HTTP response code: 500 for URL: https://api.pro.coinbase.com/products/ETH-USD/candles?granularity=60&start=2017-03-14T05:58:00Z&end=2017-03-14T10:57:00Z


For me the API returns the data correctly. 

Can you please call the link directly from your default browser and check if it works for you ?

https://api.pro.coinbase.com/products/ETH-USD/candles?granularity=60&start=2017-03-14T05:58:00Z&end=2017-03-14T10:57:00Z

AA
#9

Alex

27.09.2022 07:56
yes it works and this is the result:



===========================

[[1489489020,28.94,28.94,28.94,28.94,52.7608],[1489488960,28.94,28.94,28.94,28.94,4.6612],[1489488900,28.94,29,29,28.94,12.62602388],[1489488840,29,29,29,29,3.2126],[1489488780,29,29,29,29,23.3756],[1489488720,28.97,29.02,29.02,28.98,105.35293],[1489488660,28.95,29.05,28.95,28.99,70.6318],[1489488600,28.94,28.96,28.95,28.94,1.503145],[1489488540,28.8,29.08,29.01,28.8,38.79449],[1489488480,28.99,29,28.99,29,3.4155],[1489488420,28.82,29.05,29.05,28.82,7.429],[1489488360,29.04,29.05,29.04,29.04,169.1993],[1489488300,29.02,29.04,29.02,29.04,15.6625],[1489488240,28.95,29.05,28.95,29.05,191.8818],[1489488180,28.82,28.96,28.82,28.96,9.1223],[1489488120,28.69,28.86,28.86,28.69,85.8775],[1489488060,28.82,28.98,28.82,28.82,149.180985],[1489488000,28.75,28.82,28.75,28.82,9.430813],[1489487940,28.75,28.8,28.78,28.8,13.4808],[1489487880,28.75,28.82,28.75,28.82,30.5],[1489487820,28.74,28.79,28.78,28.79,24.734499],[1489487760,28.78,28.79,28.78,28.79,114.3162],[1489487700,28.77,28.79,28.77,28.79,19.1553],[1489487640,28.6,28.79,28.78,28.6,44.2672],[1489487580,28.73,28.79,28.79,28.79,7.115585],[1489487520,28.72,28.94,28.72,28.74,30.171018],[1489487460,28.7,28.9,28.7,28.74,140.277066],[1489487400,28.64,28.85,28.7,28.64,46.1779567],[1489487340,28.66,28.7,28.7,28.66,12.204342],[1489487280,28.62,28.84,28.62,28.67,128.443788],[1489487220,28.55,28.61,28.61,28.55,8.1616],[1489487160,28.62,28.62,28.62,28.62,5.8467],[1489487100,28.61,28.72,28.72,28.62,36.938069],[1489487040,28.72,28.72,28.72,28.72,41.9774],[1489486980,28.62,28.78,28.62,28.62,7.925877],[1489486920,28.6,28.77,28.6,28.66,203.7409],[1489486860,28.59,28.65,28.62,28.65,85.925871],[1489486800,28.59,28.61,28.61,28.59,0.9921],[1489486740,28.64,28.66,28.64,28.65,5.5428],[1489486620,28.64,28.64,28.64,28.64,67.3451],[1489486560,28.5,28.66,28.66,28.5,43.3445],[1489486500,28.66,28.69,28.67,28.66,5.81130991],[1489486440,28.66,28.74,28.66,28.71,40.9684],[1489486380,28.63,28.68,28.63,28.68,13.8998],[1489486320,28.53,28.64,28.64,28.55,29.530975],[1489486260,28.57,28.67,28.62,28.57,40.919731],[1489486200,28.56,28.78,28.65,28.67,173.152853],[1489486140,28.6,28.74,28.61,28.74,23.887095],[1489486080,28.56,28.68,28.58,28.68,24.046001],[1489486020,28.56,28.63,28.63,28.56,127.277204],[1489485960,28.58,28.67,28.58,28.6,273.761053],[1489485900,28.56,28.62,28.56,28.62,9.088779],[1489485840,28.51,28.65,28.65,28.51,23.3646],[1489485780,28.56,28.72,28.62,28.72,48.46758362],[1489485720,28.67,28.69,28.67,28.67,13.3672],[1489485660,28.58,28.69,28.69,28.66,58.51714075],[1489485600,28.69,28.78,28.69,28.71,127.789],[1489485540,28.67,28.81,28.7,28.67,31.66177859],[1489485480,28.63,28.8,28.7,28.78,80.439827],[1489485420,28.61,28.77,28.74,28.75,208.1826],[1489485360,28.74,28.88,28.87,28.74,76.337376],[1489485300,28.82,28.92,28.92,28.82,26.189571],[1489485240,28.89,29.02,28.95,28.89,353.53079064],[1489485180,28.91,29.06,28.91,28.99,16.5769],[1489485120,28.9,28.91,28.91,28.9,4.6633],[1489485060,28.93,29.02,28.95,28.94,16.972674],[1489485000,29.02,29.02,29.02,29.02,74.4944],[1489484940,29.01,29.01,29.01,29.01,19.6227],[1489484880,28.96,29.05,28.96,29.05,4.8677],[1489484820,28.82,28.98,28.98,28.82,4.9721],[1489484760,29,29,29,29,1.0293],[1489484700,28.95,29.08,28.95,28.95,130.774],[1489484640,28.94,28.98,28.98,28.95,6.549265],[1489484580,28.9,28.99,28.9,28.99,26.443],[1489484520,28.76,28.88,28.88,28.76,43],[1489484460,28.89,28.97,28.9,28.9,24.1038],[1489484400,28.9,28.93,28.93,28.93,8.9408],[1489484280,28.87,28.94,28.9,28.87,60.369533],[1489484220,28.88,28.95,28.88,28.95,33.310354],[1489484160,28.88,28.92,28.88,28.92,22.059673],[1489484100,28.88,28.89,28.89,28.88,97.953005],[1489484040,28.85,28.97,28.85,28.97,22.7768],[1489483980,28.84,28.85,28.84,28.85,2.0553],[1489483920,28.8,28.84,28.81,28.84,71.1714],[1489483860,28.86,28.86,28.86,28.86,1.2208],[1489483800,28.67,28.87,28.8,28.87,37.0364],[1489483740,28.5,28.9,28.5,28.9,247.42595958],[1489483680,28.49,28.5,28.5,28.5,58.343432],[1489483620,28.49,28.51,28.51,28.5,378.078],[1489483560,28.46,28.54,28.46,28.54,6.0692],[1489483500,28.45,28.45,28.45,28.45,1.2685],[1489483440,28.45,28.53,28.45,28.46,37.3466],[1489483380,28.44,28.44,28.44,28.44,6.405966],[1489483320,28.43,28.49,28.43,28.49,28.8775],[1489483260,28.4,28.5,28.4,28.48,98.67598677],[1489483200,28.4,28.44,28.4,28.44,35.9207],[1489483140,28.4,28.43,28.4,28.43,119.9401],[1489483080,28.37,28.4,28.37,28.4,179.144],[1489483020,28.32,28.4,28.38,28.32,11.838676],[1489482960,28.37,28.4,28.37,28.4,83.6121],[1489482900,28.3,28.35,28.35,28.31,362.0272],[1489482840,28.39,28.4,28.39,28.4,4.7565],[1489482780,28.36,28.39,28.36,28.39,0.9823],[1489482720,28.36,28.36,28.36,28.36,0.8439],[1489482660,28.3,28.35,28.33,28.35,17.070299],[1489482600,28.33,28.4,28.33,28.37,116.62128302],[1489482540,28.33,28.4,28.35,28.4,262.8477],[1489482480,28.38,28.38,28.38,28.38,8.7566],[1489482420,28.33,28.45,28.4,28.33,260.896204],[1489482360,28.36,28.49,28.44,28.46,80.175121],[1489482300,28.42,28.49,28.49,28.42,8.33252059],[1489482240,28.48,28.52,28.48,28.52,29.2624],[1489482180,28.34,28.51,28.51,28.49,134.6914],[1489482120,28.5,28.58,28.58,28.5,30.2659],[1489482060,28.58,28.58,28.58,28.58,0.2782],[1489482000,28.57,28.6,28.6,28.57,11.582569],[1489481940,28.5,28.64,28.64,28.5,41.3117],[1489481880,28.61,28.7,28.61,28.61,67.561856],[1489481820,28.54,28.72,28.65,28.54,120.69246123],[1489481760,28.65,28.7,28.65,28.7,100.6941],[1489481700,28.65,28.71,28.71,28.66,29.996258],[1489481640,28.7,28.8,28.75,28.77,93.71163254],[1489481580,28.75,28.81,28.81,28.75,5.4803687],[1489481520,28.63,28.83,28.83,28.82,104.08169678],[1489481460,28.81,28.89,28.81,28.89,95.3797],[1489481400,28.77,28.8,28.8,28.77,10.913388],[1489481340,28.8,28.82,28.82,28.81,27.71468],[1489481280,28.8,28.84,28.84,28.8,7.227939],[1489481220,28.84,28.84,28.84,28.84,97.2797],[1489481160,28.86,28.87,28.86,28.86,8.2928],[1489481100,28.83,28.87,28.83,28.87,5.384295],[1489481040,28.75,28.9,28.85,28.9,22.8043],[1489480980,28.86,28.91,28.86,28.9,21.6318],[1489480920,28.86,28.86,28.86,28.86,27.8395],[1489480860,28.86,28.86,28.86,28.86,104.6347],[1489480800,28.86,28.86,28.86,28.86,15.6148],[1489480740,28.82,28.89,28.83,28.89,156.7655],[1489480680,28.83,28.83,28.83,28.83,1.0579],[1489480620,28.82,28.83,28.82,28.83,36.9553],[1489480560,28.75,28.84,28.81,28.75,62.13817493],[1489480500,28.8,28.86,28.8,28.86,82.1438],[1489480380,28.8,28.8,28.8,28.8,1.5724],[1489480320,28.75,28.84,28.84,28.83,10.3448],[1489480260,28.85,28.85,28.85,28.85,3.5802],[1489480200,28.85,28.85,28.85,28.85,2.1938],[1489480140,28.8,28.87,28.83,28.87,4.82214534],[1489480080,28.87,28.87,28.87,28.87,3.1231],[1489480020,28.87,28.87,28.87,28.87,50.4619],[1489479960,28.87,28.87,28.87,28.87,54.134],[1489479900,28.74,28.93,28.92,28.74,523.257647],[1489479840,28.93,28.94,28.93,28.93,8.5536],[1489479780,28.91,28.94,28.91,28.94,313.8823],[1489479720,28.92,28.92,28.92,28.92,19.2341],[1489479660,28.91,28.91,28.91,28.91,74.2384],[1489479600,28.91,28.91,28.91,28.91,10.8617],[1489479480,28.82,28.92,28.82,28.91,144.8933],[1489479420,28.77,28.82,28.82,28.77,10.4901],[1489479360,28.67,28.9,28.9,28.85,546.8727141],[1489479300,28.89,28.9,28.9,28.89,152.294479],[1489479240,28.89,28.9,28.89,28.9,24.8578],[1489479180,28.89,28.89,28.89,28.89,1.6101],[1489479120,28.82,28.9,28.86,28.9,12.4739],[1489479060,28.89,28.92,28.89,28.92,35.9881],[1489479000,28.87,28.9,28.87,28.9,112.1474],[1489478940,28.79,28.89,28.82,28.87,289.0558],[1489478880,28.74,28.89,28.8,28.89,23.8665],[1489478820,28.81,28.9,28.81,28.9,2.8371],[1489478760,28.84,28.84,28.84,28.84,1.7243],[1489478700,28.81,28.88,28.81,28.88,35.209],[1489478640,28.56,28.92,28.9,28.81,463.95603],[1489478580,28.91,28.93,28.91,28.91,17.7466],[1489478520,28.9,28.91,28.9,28.91,16.4642],[1489478460,28.86,28.93,28.86,28.86,45.55469577],[1489478400,28.86,28.92,28.9,28.9,28.413222],[1489478340,28.88,28.92,28.88,28.9,32.4678],[1489478280,28.85,28.89,28.85,28.89,3.6836],[1489478220,28.85,28.89,28.88,28.85,91.071582],[1489478160,28.84,28.89,28.86,28.84,27.5451687],[1489478100,28.8,28.94,28.87,28.94,158.510381],[1489478040,28.86,28.94,28.93,28.86,103.88629248],[1489477980,28.89,28.94,28.92,28.94,77.7235],[1489477920,28.86,28.91,28.91,28.86,22.2155],[1489477860,28.93,28.94,28.94,28.93,39.83084338],[1489477800,28.92,29.11,28.96,28.94,57.70358677],[1489477740,28.96,28.97,28.97,28.96,21.24562],[1489477680,28.97,29.03,29.03,29,42.0563],[1489477620,28.96,29.12,29.01,29.02,18.052933],[1489477560,29.01,29.01,29.01,29.01,4.1494],[1489477500,28.94,29.15,28.97,28.94,270.0386213],[1489477440,28.88,29.12,28.97,29.12,92.085988],[1489477380,28.89,29.11,28.99,29.02,44.0499],[1489477320,28.88,29.04,28.92,29.04,58.578612],[1489477260,28.85,29.08,29.06,28.92,235.36080047],[1489477200,29.04,29.12,29.08,29.04,111.84790981],[1489477140,29.06,29.15,29.09,29.12,94.53525988],[1489477080,29.06,29.12,29.12,29.06,145.983265],[1489477020,29.06,29.16,29.12,29.16,446.45654],[1489476960,29.06,29.15,29.14,29.06,95.063109],[1489476900,29.06,29.15,29.15,29.06,284.37036974],[1489476840,29.11,29.27,29.13,29.11,102.754604],[1489476780,29.1,29.13,29.11,29.13,73.878],[1489476720,29.11,29.32,29.18,29.11,98.411727],[1489476660,29.19,29.19,29.19,29.19,0.3412],[1489476600,29.1,29.21,29.18,29.21,76.3707],[1489476540,29.22,29.25,29.22,29.25,8.0468],[1489476480,29.11,29.24,29.23,29.24,33.5164],[1489476420,29.13,29.25,29.25,29.24,46.99436861],[1489476360,29.2,29.35,29.25,29.26,385.113979],[1489476300,29.2,29.33,29.33,29.33,361.049333],[1489476240,29.29,29.35,29.29,29.35,100.00040847],[1489476180,29.24,29.35,29.24,29.35,225.3812],[1489476120,29.1,29.3,29.12,29.3,79.78912411],[1489476060,29.15,29.16,29.15,29.16,6.7883],[1489476000,29.06,29.18,29.15,29.16,183.689452],[1489475940,29.12,29.33,29.13,29.15,232.88082],[1489475880,29.11,29.12,29.11,29.12,7.5038],[1489475820,29.1,29.11,29.1,29.11,1.669],[1489475760,29.02,29.13,29.09,29.12,24.7229],[1489475700,29.05,29.16,29.05,29.16,68.3746],[1489475640,29.01,29.04,29.04,29.01,7.0396],[1489475580,29.01,29.06,29.05,29.01,45.1322],[1489475520,29.02,29.19,29.06,29.08,136.58598],[1489475460,29.07,29.13,29.12,29.07,11.68386],[1489475400,29.06,29.15,29.06,29.15,46.05128439],[1489475340,29.04,29.12,29.12,29.04,3.61735],[1489475280,29,29.1,29.03,29.1,63.107693],[1489475220,29.03,29.3,29.2,29.03,216.8418292],[1489475160,29.1,29.29,29.1,29.29,135.0076],[1489475100,29.02,29.11,29.02,29.11,38.838594],[1489475040,29.02,29.02,29.02,29.02,10.3083],[1489474980,29.01,29.02,29.02,29.01,8.66913],[1489474920,28.93,29.02,28.95,29.02,227.217402],[1489474860,28.91,29.03,28.95,28.99,96.0602126],[1489474800,28.92,28.95,28.93,28.95,10.7008],[1489474740,28.87,29.03,29.03,28.87,37.8799],[1489474680,29.03,29.03,29.03,29.03,34.9041],[1489474620,28.88,29.24,29.01,29.03,126.152088],[1489474560,28.98,29.25,29,29.05,427.182405],[1489474500,28.96,29.02,28.96,28.97,172.22060328],[1489474440,28.82,29.06,29.06,28.98,17.7693],[1489474380,28.98,29.1,28.98,29.09,345.2347],[1489474320,28.85,28.99,28.99,28.9,12.3645],[1489474260,29,29,29,29,9.7022],[1489474200,28.87,29.05,29.05,28.87,39.051222],[1489474140,29.08,29.08,29.08,29.08,34.8901],[1489474080,28.94,29.14,29.13,29.05,419.200301],[1489474020,29.17,29.17,29.17,29.17,15.848],[1489473960,29.13,29.2,29.16,29.19,38.306],[1489473900,29.15,29.23,29.23,29.21,16.27948179],[1489473840,29.13,29.29,29.25,29.29,206.7328],[1489473780,29.24,29.3,29.24,29.3,55.1161],[1489473720,29.22,29.29,29.24,29.23,19.267725],[1489473660,29.29,29.3,29.29,29.3,196.2233],[1489473600,29.21,29.29,29.21,29.29,12.79452462],[1489473540,29.2,29.23,29.23,29.22,13.1322],[1489473480,29.28,29.28,29.28,29.28,5.7251],[1489473420,29.11,29.29,29.26,29.29,3.01],[1489473360,29.29,29.3,29.3,29.3,50.4556],[1489473300,29.19,29.33,29.2,29.3,38.7842],[1489473240,29.11,29.34,29.34,29.11,47.1414],[1489473180,29.34,29.35,29.35,29.34,169.3147],[1489473120,29.35,29.35,29.35,29.35,4.2043],[1489473060,29.33,29.35,29.35,29.35,5.795649],[1489473000,29.34,29.35,29.34,29.34,177.5874588],[1489472940,29.34,29.35,29.34,29.34,42.72702849],[1489472880,29.34,29.34,29.34,29.34,6.8951],[1489472820,29.33,29.35,29.34,29.34,82.2371],[1489472760,29.23,29.35,29.23,29.35,235.5709],[1489472700,29.07,29.31,29.31,29.07,372.7628],[1489472640,29.27,29.35,29.27,29.35,26.1778],[1489472580,29.17,29.27,29.21,29.27,44.4664],[1489472520,29.14,29.33,29.29,29.14,49.681325],[1489472460,29.14,29.3,29.29,29.14,115.430208],[1489472400,29.25,29.34,29.25,29.34,10.4108],[1489472340,29.14,29.36,29.14,29.23,768.20876396],[1489472280,29.11,29.19,29.12,29.19,20.226],[1489472220,29.12,29.19,29.12,29.19,171.542],[1489472160,29.06,29.13,29.12,29.06,37.2481],[1489472100,29.06,29.08,29.08,29.06,11.3719],[1489472040,29.05,29.13,29.06,29.13,20.0071],[1489471980,29.09,29.12,29.09,29.12,72.94],[1489471920,28.67,29.15,28.97,29.15,1108.379983],[1489471860,28.97,29.08,29.08,28.97,76.6168],[1489471800,29,29.16,29.07,29.16,50.6078],[1489471740,29.08,29.32,29.2,29.08,68.50928215],[1489471680,29.2,29.61,29.39,29.2,551.3416746],[1489471620,29.33,29.51,29.43,29.38,168.0063],[1489471560,29.41,29.54,29.41,29.43,152.90351],[1489471500,29.4,29.45,29.4,29.45,8.371203],[1489471440,29.45,29.49,29.45,29.48,1.3007],[1489471380,29.3,29.55,29.55,29.54,107.6121],[1489471320,29.5,29.62,29.61,29.5,205.12591497],[1489471260,29.6,29.7,29.64,29.6,119.5202],[1489471200,29.42,29.69,29.46,29.67,246.702296],[1489471140,29.41,29.47,29.47,29.41,99.693337],[1489471080,29.27,29.33,29.32,29.27,129.5227]]

AA
#10

Alex

27.09.2022 07:57

Attachment Screenshot 2022-09-26 154607.png added

Screenshot 2022-09-26 154607.png
(20.13 KiB)
but this is what was shown in SQX page.
TT
#11

Tamas

27.09.2022 08:02

Status changed from Waiting for information to New

In the log, is there again the same error ? We will test it out.
AA
#12

Alex

27.09.2022 08:27

Attachment Screenshot 2022-09-27 090830.jpg added

Attachment Screenshot 2022-09-27 091629.png added

Attachment log_2022_09_27.log added

Attachment Screenshot 2022-09-27 092453.png added

Attachment log_2022_09_27.log added

Screenshot 2022-09-27 092453.png
(48.42 KiB)
log_2022_09_27.log
(30.29 KiB)
Screenshot 2022-09-27 090830.jpg
(233.65 KiB)
Screenshot 2022-09-27 091629.png
(52.90 KiB)
log_2022_09_27.log
(17.38 KiB)
Again Occurred:

And another thing, when we download, and during download error occurs, SQX does not show latest downloaded data.

but when we download again and stop soon, they appear.

g
#13

Lee Guan Chuan

29.09.2022 08:18

Status changed from New to Fixed

Hi Alex,


  1. I have updated the system to update the data and show the correct date when there is an error happens while downloading.
  2. I have updated the system to display the error message. You just need to move your mouse to the "ERROR" text.
  3. The download error was caused by the API from Coinbase. This is where I couldn't solve it, if there is an error from Coinbase API, you can try again later when Coinbase API is stable.

The fixes above should release in the DEV5.


Thank you.

AA
#14

Alex

29.09.2022 09:48
Dear Lee,

I have an offer.

It seems I have different data every time I download data from coinbase and binance.

And it can be from API unresponsiveness of the exchanges.

By the way it will be very nice that SQX handles this errors, and results in a fine and robust data.

e.g. when data ends with error in coinbase, I do not have correct downloaded ones.

please take a look and see if it is possible.

Thank you.

g
#15

Lee Guan Chuan

29.09.2022 10:07

Status changed from Fixed to In progress

Hi Alex, 


Thanks for letting us know. Surely I will take a look into this. I will mark this task back to pending for this issue.


Thank you.


Votes: +2

Drop files to upload

or

choose files

Max size: 5MB

Not allowed: exe, msi, application, reg, php, js, htaccess, htpasswd, gitignore

...
Wait please