This seems like a use case we would want targets t...
# singer-target-development
t
This seems like a use case we would want targets to generally support. May be hard to generalize for the SDK though. Thoughts <!subteam^S02BCD9FFEF>?
e
I agree that targets/databases that can do some equivalent of
alter table my_table add column ...
should support schemas with added fields. Maybe target-athena can be a reference implementation for the sdk?
t
This could be a good capability to document too
k
Bigquery is a bit of a weird case too. It supports adding columns but with limitations. You cannot add a new REQUIRED field, it must be NULLABLE or REPEATED, and the error it throws according to the docs is similar to the one posted. I haven’t checked, but maybe the target doesn’t even try to resolve the conflict. The only other alternative is to copy into a new table, but that messes up any partitions. BQ is not the funnest 😅 You know it’s bad when there are official workarounds.
a
Yeah, I'd very much like to build this into the SDK. I imagine there's something like
Sink.prep_destination(stream_name, schema, key_properties)
and for tabular/columnar Sinks, a per-column, something like
ensure_column_exists(...)
and
ensure_column_compatible(...)
. Even a
TabularSink.get_native_type(jsonschema_type) -> Any
might be helpful because we can catch errors and give a developer-provided fallback type automatically.