Hey everyone! Regarding the current target-bigquery by @alexander_butler, has anyone had issues regarding the "overwrite" parameter? We would like to reproduce the "truncate" interaction from typical dbs, but are getting weird interactions where the table is set to expire every 24h. Any idea how we could remove this? It's causing issues in production. Thanks!

Hey! 👋 So the only place an expiry is set is here for merge/overwrite https://github.com/z3z1ma/target-bigquery/blob/main/target_bigquery/core.py#L334 https://github.com/z3z1ma/target-bigquery/blob/main/target_bigquery/core.py#L316 The table that is actually getting set to expire is an intentional temp table. That is where records are written and it should be something like

<stream_name>__<timestamp>

Both that temp table and a permanent table should be created and both should exist. It is a subsequent SQL operation post-load that does the overwrite. This is done to achieve atomicity. It would be pretty lame to truncate someones table and then their tap fails. So thats the deal.

Hey @alexander_butler, thanks for the answer! I just ran the target with these parameters:

denormalized: false
      method: batch_job
      timeout: 300
      batch_size: 250000
      append_columns: true
      generate_view: false
      overwrite: true
      fix_columns:
        lower: False
        quotes: False
        add_underscore_when_invalid: True

And got a non-timestamp table that has an expiration date. Is this normal?

Can you give it another shot on the main branch

Ouh I'll give it a try, thanks!

Awesome, I'll cut a patch release later today. Thanks @Stéphane Burwash 🎉

Hey @alexander_butler, created closed issue 😄 https://github.com/z3z1ma/target-bigquery/issues/55