I saw that Gitlab is using tap-zoom in an <@U069EP...
# singer-tapss
stephen_lloyd
05/07/2021, 12:32 PMI saw that Gitlab is using tap-zoom in an @taylor post or recording somewhere recently. I pulled down the mashey/tap-zoom and have been running it. I think I never should have looked at the log output because it seems agonizing. I’m using pipelinewise’s snowflake target. I looked through the API and see that there’s lots of nested calls. My question: how long is it taking for your team to run the tap?
stephen_lloyd
05/07/2021, 12:39 PMSecond question, has anyone discussed making this an incremental tap?
t
taylor
05/07/2021, 1:58 PMI haven’t touched that tap in months. @vprakash do you have any insight on this?
taylor
05/07/2021, 1:58 PMThe Zoom API doesn’t support incremental replication as I understand it https://github.com/Mashey/tap-zoom#data-replication
v
vprakash
05/07/2021, 1:59 PMI was working on it one week back. The tap run endlessly if the the volume is high like in more than 10K records . Still working on next step of action. One action item was to reduce the snowflake buffer value to increase the performance of the tap.
s
stephen_lloyd
05/07/2021, 2:08 PM@vprakash you were working on incremental loading or performance in general?
stephen_lloyd
05/07/2021, 2:09 PMIs there any performance difference using the datamill snowflake target?
v
vprakash
05/07/2021, 2:09 PMI was working on performance in general.