I am trying extract and load a month's data from

tap-zendesk

to

target-postgres

. It is running for almost 24 hrs now and according to logs, its still extracting the data but I don't see any data yet loaded in the database. Can you please let me know if there is a way we can configure to extract and load continuously for large datasets? Also, what happens to the pipeline if restart the server now?

hmm long running tap. It would depend on how target-postgres is loading the data. Which target-posgres are you using will help debug this, and the settings you're using probably matter. I thought most of the target db's upsert over time with postgres but I'm not sure

Hi Derek, I am using following variant of target-postgres. https://github.com/datamill-co/target-postgres

Hi @sujit! That variant denests/flattens objects by default and it can't be turned off. I'm not familiar with zendesk, so I don't know if the objects might be too nested, causing the target to spend a lot of time walking through nested lists and dicts. If you wanna make sure data will land in your target eventually, you can try with a lower max_batch_rows setting

yep, a number of zendesk objects have nested properties, occasionally a couple levels

Thanks for suggestions, Edgar and Prratek

@sujit you can try the transferwise variant

Thanks, Edgar.

@jo_pearson think we have tried this but wanted to tag you in thread