Does iceberg always require `add_record_metadata: ...
# getting-started
t
Does iceberg always require
add_record_metadata: true
to take care of snapshots? If I have those extra
_sdc_
columns, I get
No schema for record field
error for these columns: "_sdc_deleted_at" and "_sdc_table_version" (when there are null values in those columns)
e
I'm not familiar with iceberg or target-iceberg, but https://github.com/meltano/sdk/blob/3ea0667b982af3efe514185e865951c5bd8c7251/singer_sdk/sinks/core.py#L167-L168 should ensure those are present in the schema.
t
@Edgar Ramírez (Arch.dev) Thanks. Yes they are present, by default. I'm wondering if it is safe to remove those columns. Maybe someone with the experience of iceberg table format would be able to confirm if these columns are responsible for the
snapshot
feature.
e
I think you're the maintainer of target-iceberg? If so, you probably know best whether the snapshot implementation in the target would benefit from using those columns 😅
t
@Edgar Ramírez (Arch.dev) I'm one of the maintainers. Unfortunately, I'm still learning more and more about iceberg and the other table formats. Maybe I can find a work around soon.
👍 1