When I add a column to a stream, do I need to run ...
# best-practices
i
When I add a column to a stream, do I need to run a full refresh of that tap or does meltano recognize that new column and do a full refresh of that particular stream to whichever target I'm loading it to? How do you guys manage adding a new column to a stream if your tap/pipeline is running in a container in production? Do you like delete the table on the database side then have to go and run a full refresh for that tap any time you make a single column change to one of your streams?
e
The target would normally recognize the new column and update the destination table, but beyond that nothing else is bubbled up to Meltano and you need to manually run a full-refresh
1
i
Does running a full refresh tell meltano to truncate or recreate the target table or does that need to be done manually?
e
Not necessarily, depending on the loader's built-in or configurable strategy, it will either upsert, append or truncate the destination tables.
i
@Edgar Ramírez (Arch.dev) If I add a new column to a stream, say "customer_address", and I have a primary key in my existing downstream table with a primary key of "customer_id" which is populated in all rows for the full range of data already, and I pass a full refresh flag for the tap, will it upsert data such that it adds that new column to existing rows where customer_id = customer_id?