Is there a way to allow any arbitrary fields come ...
# troubleshooting
m
Is there a way to allow any arbitrary fields come through via
schema
? Say fields could change at any given run. I would want to log it as a json, then within the loader flatten it to create a null column if that field didn’t exist in the past.
Currently I’ve worked around it by just adding it to a
data
field like this:
Copy code
schema = th.PropertiesList(  # dynamic number of incoming columns
            th.Property("timestamp", th.DateTimeType, required=True),
            th.Property("data", th.AnyType, required=True)
        ).to_dict()
Then in get_records
Copy code
yield {'data': record, self.replication_key: record['replication_key']}