From the office hours discussion about target functionality 🧵
pat_nadolny
06/21/2023, 5:02 PM
Related to having a default config option to either append or upsert with potentially append being the default. I thought airbyte had an option like this but it actually was related to schema changes in https://airbyte.com/blog/airbyte-schema-propagation-keeping-your-replicated-catalog-up-to-date. This is likely another default config that we'd want to add to all target, right now its a setting that developers chose in the connector but isnt configurable by default. The pipelinewise implementation details are https://transferwise.github.io/pipelinewise/user_guide/schema_changes.html. I think right now the SDK will always attempt to alter the column if the schema changes and fail if its unable to, columns that are dropped from the schema will not get dropped from the destination table.
Hi Pat, these documentation changes sound great for me. Actively working on a Clickhouse Target, got the MVP into production this morning (woo), so want to start working on understanding the SQL Target a little more as I haven't had a chance to get deep into the wiring just yet 🙂