Hello awesome people, I am using a custom meltano loader for Iceberg (Loader: https://github.com/SidetrekAI/target-iceberg) But when I tried to run an ingestion job extracting data from postgresql table and writing/loading data in iceberg format, I found an interesting problem! When the postgresql table has a column where there is an empty dictionary, I see this:

key: metadata has no fields defined, this may cause saving parquet failure as parquet doesn't support empty/null complex types [array, structs]

I tried changing the type to

pa.string

and even

pa.null

but that produced a type mismatch with singer schema. In postgresql, it is a

dict

which is empty. But while trying to write the parquet file, empty structs is not supported. How would you handle this kind of issue?

I think I would handle variant

object

types by serializing them as JSON strings and storing them as

pa.string

.

Thanks. I'll try this