From the office hours discussion about target func...
# singer-target-development
p
From the office hours discussion about target functionality 🧵
Related to having a default config option to either append or upsert with potentially append being the default. I thought airbyte had an option like this but it actually was related to schema changes in https://airbyte.com/blog/airbyte-schema-propagation-keeping-your-replicated-catalog-up-to-date. This is likely another default config that we'd want to add to all target, right now its a setting that developers chose in the connector but isnt configurable by default. The pipelinewise implementation details are https://transferwise.github.io/pipelinewise/user_guide/schema_changes.html. I think right now the SDK will always attempt to alter the column if the schema changes and fail if its unable to, columns that are dropped from the schema will not get dropped from the destination table.
cc @ken_payne @taylor @edgar_ramirez_mondragon
Some new issues: • feat: configurable target schema change settings by default - https://github.com/meltano/sdk/issues/1781 • document default sql target schema change logic - https://github.com/meltano/sdk/issues/1782 • SQL targets should allow configurable insert vs upsert behavior by default - https://github.com/meltano/sdk/issues/1783
And I think @ken_payne is creating at least one more issue in the SDK repo related to https://github.com/MeltanoLabs/target-snowflake/issues/61
b
Hi Pat, these documentation changes sound great for me. Actively working on a Clickhouse Target, got the MVP into production this morning (woo), so want to start working on understanding the SQL Target a little more as I haven't had a chance to get deep into the wiring just yet 🙂