Hi! I am working on expanding our ELT pipelines fo...
# best-practices
a
Hi! I am working on expanding our ELT pipelines for all external systems through meltano and I ran into a question around best practices. Question: Is there a recommended way to have some tables that need a full refresh and some that need incremental updates for the same tap? • For example, this seems to be easy for salesforce as I can choose for each object within the metadata • I am having a harder time with zendesk for example (cannot describe the type of extract through the metadata, at least that is what it seems like to me) ◦ Thinking if I need to make 2 taps, one with the objects I want incremental and the other with the objects I want full-refresh for? Thanks!
t
Are you saying you’re setting the metadata for a stream via https://meltano.com/docs/plugins.html#extractors and it’s not working for tap zendesk?
a
yes
I set full_table like below (this is how I have done it for salesforce) but if I don't specify in the
meltano elt
command, it uses the last checkpoint.
Also this might all be absolutely wrong so sorry if I messed up!
t
it’s possible the tap isn’t respecting
replication-method
https://github.com/twilio-labs/twilio-tap-zendesk/blob/master/tap_zendesk/streams.py#L192
a
Ooo will check today!
Update on this, have tried forced, re-installed the tap and only used forced, changed both YAML only vs set through meltano CLI, no luck. Even with forced-replicated-method (tried forced-replication-method found in some previous documentation for zendesk) it does not work