Small doubt, what does

full refresh

imply?

• A

--full-refresh

flag can be passed to perform a full refresh, ignoring state left behind by any previous runs with the same State ID.

Does that mean a full table replication?

No it means no state will be sent to the tap.

Can you please give a brief on that subtle difference? Basically I was trying to see if there’s a way to do incremental replication every 10 minutes, but also a full table truncate replication once a day.

I'm not sure what you mean https://meltano.slack.com/archives/C01TCRBBJD7/p1669232150508929?thread_ts=1669230939.620099&cid=C01TCRBBJD7 is exactly what is happening 😞 not sure how else to explain

Subtly distinction: • "FULL_TABLE" is a mode per-stream that doesn't attempt incremental replication. • "Not using prior state" means that we'll run the extract operation as if if were the first time. I can't think of any real behavioral differences in these two though...

You guys are amazing! Thanks, @visch @aaronsteers!

Thanks, @abhishek_ajmera. Much appreciated. 🙂 If you take another look at that issue description, I just finished adding some nuance there. It's good for us to think through this. Also - we've had something like 10 references to

ACTIVATE_VERSION

requirements just in Slack this month so I'm spinning off a #C04CAST0E20 channel for more discussion. Right now the channel is empty, but feel free to join if this is a topic you are interested in.

I wish I could be of help but I can’t say I have enough knowledge to think about the nuances mentioned or any other. 😅

Thanks 🙂

Oh no, thanks but I already did a short term fix. I used

target-bigquery

which has a configuration for the way it inserts data, truncate for FULL_TABLE and append for INCREMENTAL. Was using two environments to replicate some tables fully and some tables incrementally, so just added the incremental tables to the list of first env as well. 😬

Nice! Glad you got a workaround 👍