Question: if a stream has a pre-defined schema and one of the columns/fields is no longer returned by the source (eg column deleted in the source database, REST api no longer returns the field, etc.) and the stream's schema is not updated, what happens?
v
visch
12/05/2024, 3:02 PM
Record doesn't have that key sent as it's missing
s
steven_wang
12/05/2024, 3:07 PM
Got it, would an error be thrown or no?
steven_wang
12/05/2024, 3:07 PM
Just curious if it would break any existing pipelines
e
Edgar Ramírez (Arch.dev)
12/06/2024, 12:09 AM
If the schema says it's a required field, it will fail validation (in targets that perform schema validation), otherwise the column will be created but no data will be loaded to it.