I have a question about incremental loading and ho...
# best-practices
j
I have a question about incremental loading and how taps and targets should behave as part of the protocol. I'm using for example the target https://hub.meltano.com/loaders/target-bigquery/ (adswerve variant). Many tap supports incremental loading for some streams, but not for all. The bigquery-target has a static configuration to either load with APPEND (incremental) or TRUNCATE (full load). At the moment I always have to split my jobs and use TRUNCATE by default, and only use APPEND for tables that support incremental loading. This is super cumbersome however. Isn't there a way that taps and targets can communicate automatically how each stream should be loaded?
v
Yes, https://hub.meltano.com/singer/docs#activate-version Normally targets need to assume it's incremental loads unless activate version tells them otherwise
For bigquery I'd highly recommend you look at https://hub.meltano.com/loaders/target-bigquery--z3z1ma/
I don't know if that target supports activate version though still a nice target!
j
thanks for the input @visch!