Question on loading data: If a column exists in a...
# singer-targets
s
Question on loading data: If a column exists in a source table but is entirely empty, what is the expected behaviour in the warehouse target? Would the column be created and populated with nulls, or would it not be created? And if so, where does the responsibility fall for making this decision: in the loader, or in the warehouse in itself? I'm trying to write dbt schemas for incoming data, but cannot complete them because some columns are not appearing in my warehouse
t
I think it would be wrong for a tap to make judgments about whether a column should be included for any reason... if you've asked to have it included in your config it should be included. 🤷🏻‍♂️ Similarly, if the stream from the tap includes a certain column I would think the target is obliged to create it. FWIW your problem sounds more like a stale catalog problem to me than anything to do with empty columns, but... that could just be because I get bit by stale cache issues all the time. 😉
s
Agreed. Guess this is another reason why we will be moving away from stitch; they don't include a column if it doesn't exist. Quite aggravating