matt_elgazar
10/23/2023, 8:49 PMschema
? Say fields could change at any given run. I would want to log it as a json, then within the loader flatten it to create a null column if that field didn’t exist in the past.matt_elgazar
10/23/2023, 10:45 PMdata
field like this:
schema = th.PropertiesList( # dynamic number of incoming columns
th.Property("timestamp", th.DateTimeType, required=True),
th.Property("data", th.AnyType, required=True)
).to_dict()
Then in get_records
yield {'data': record, self.replication_key: record['replication_key']}