Parent/Child relationship question,

time=2021-08-31 15:54:32 name=tap-clickup level=WARNING message=Property 'team_id' was present in the 'space' stream but not found in catalog schema. Ignoring.

it looks like the team_id gets added into my space stream because Team is a parent of Space in this case. Is the idea that the tap developed should add team_id to the schema for the data coming from the space object? Seems a bit odd as

team_id

isn't in the data that comes from the space api query. Maybe I'm missing something, thought I'd ask first before throwing in an issue

Hi, @visch. Yes, I think by default the parent context gets added to the child stream - the logic is actually inherited (and was preserved) from the partition use cases. The idea is the parent context (and/or partition context) is relevant for audit and debugging context.

By default why not add the property to the schema then? Hmm I understand the debugging want, but I don't like the idea of adding things that are not from the actual source. A convention like sdc_space_id or something like sounds like the best of both worlds

By default why not add the property to the schema then?

Perhaps we should! I'm not sure which approach is better, tbh.

Hmm I understand the debugging want, but I don't like the idea of adding things that are not from the actual source. A convention like sdc_space_id or something like sounds like the best of both worlds

Certainly I think there's space to refine and improve - I don't love an infinite/unbound count of

_sdc_

columns but perhaps a single

_sdc_partition_context

could hold a complex dict of the parent context, and/or something like

_sdc_parent_context

. Would be a much appreciated issue topic and/or office hours topic. 🙂

On one extreme you could load all the code needed for the tap into one of those fields