and in that entire time it didn’t upload any recor...
# troubleshootingc
chris_kings-lynne
05/11/2021, 5:23 AMand in that entire time it didn’t upload any records to redshift. it’s like it’s waiting for the entire data set to be downloaded before starting any uploading?
x
xinge_li
05/11/2021, 2:00 PMexactly as what I saw, it doesn’t load anything to my postgres 🥲 I got the same feeling
t
taylor
05/11/2021, 2:53 PMAre you seeing any records output on the command line?
d
douwe_maan
05/11/2021, 3:08 PM@chris_kings-lynne You may want to tweak this setting: https://meltano.com/plugins/loaders/redshift.html#batch-size-rows
c
chris_kings-lynne
05/12/2021, 1:25 AMThis is a tiny sample of a 13 hour run that failed when it went OOM. During this period, exactly zero rows were written to Redshift. It also seems to spend like an hour to extact 100 records
chris_kings-lynne
05/12/2021, 1:26 AM@douwe_maan no point really tweaking that as if the run doesn’t finish the state won’t be written and that data in redshift doesn’t help us right
t
taylor
05/12/2021, 1:20 PMI’ve tested tap-zendesk locally before and it seemed to work with a smaller set of data - I think I was extracting just a month’s worth of tickets. It sounds like it’s something with the tap just not sending records.
c
chris_kings-lynne
05/13/2021, 1:39 AMI’ve still not managed to get a 12 month refresh to complete
chris_kings-lynne
05/13/2021, 1:39 AMlatest failure after 4 hours is:
chris_kings-lynne
05/13/2021, 1:40 AMUntitled
chris_kings-lynne
05/13/2021, 1:40 AMI think at this point (a) the zendesk api is insanely slow/rate limited
chris_kings-lynne
05/13/2021, 1:41 AMand (b) the tap is not coded defensively against running for a very long time
chris_kings-lynne
05/13/2021, 1:45 AMWith that run I was using redshift flush all streams at 500 records, and I am getting rows in redshift
d
douwe_maan
05/13/2021, 2:15 PM@chris_kings-lynne So target-redshift does not output state until the very end? Or tap-zendesk itself doesn't? We should address that so that even an interrupted run can be recovered
c
chris_kings-lynne
05/14/2021, 1:35 AMIt does seem to update state now after each batch upload, so we’re good. the problem was that in a 14 hour run, tap zendesk never got to the 100k default rows for redshift