Hi! I am working on expanding our ELT pipelines fo...
# best-practicesa
aditya_goyal
01/11/2022, 7:16 PMHi! I am working on expanding our ELT pipelines for all external systems through meltano and I ran into a question around best practices.
Question:
Is there a recommended way to have some tables that need a full refresh and some that need incremental updates for the same tap?
• For example, this seems to be easy for salesforce as I can choose for each object within the metadata
• I am having a harder time with zendesk for example (cannot describe the type of extract through the metadata, at least that is what it seems like to me)
◦ Thinking if I need to make 2 taps, one with the objects I want incremental and the other with the objects I want full-refresh for?
Thanks!
t
taylor
01/12/2022, 9:00 PMAre you saying you’re setting the metadata for a stream via https://meltano.com/docs/plugins.html#extractors and it’s not working for tap zendesk?
a
aditya_goyal
01/13/2022, 12:21 AMyes
aditya_goyal
01/13/2022, 12:25 AMI set full_table like below (this is how I have done it for salesforce) but if I don't specify in the
meltano eltaditya_goyal
01/13/2022, 12:26 AMAlso this might all be absolutely wrong so sorry if I messed up!
t
taylor
01/13/2022, 4:03 PMit’s possible the tap isn’t respecting
replication-methodtaylor
01/13/2022, 4:05 PMhttps://github.com/twilio-labs/twilio-tap-zendesk#sideloading-for-tickets maybe trying
forced-replicated-methoda
aditya_goyal
01/13/2022, 4:26 PMOoo will check today!
aditya_goyal
01/25/2022, 4:33 PMUpdate on this, have tried forced, re-installed the tap and only used forced, changed both YAML only vs set through meltano CLI, no luck.
Even with forced-replicated-method (tried forced-replication-method found in some previous documentation for zendesk) it does not work