Parent/Child relationship question, `time=2021-08-...
# singer-tap-development
v
Parent/Child relationship question,
time=2021-08-31 15:54:32 name=tap-clickup level=WARNING message=Property 'team_id' was present in the 'space' stream but not found in catalog schema. Ignoring.
it looks like the team_id gets added into my space stream because Team is a parent of Space in this case. Is the idea that the tap developed should add team_id to the schema for the data coming from the space object? Seems a bit odd as
team_id
isn't in the data that comes from the space api query. Maybe I'm missing something, thought I'd ask first before throwing in an issue
a
Hi, @visch. Yes, I think by default the parent context gets added to the child stream - the logic is actually inherited (and was preserved) from the partition use cases. The idea is the parent context (and/or partition context) is relevant for audit and debugging context.
You are free to add the parent context key to schema (recommended) or ignore the warning, or you can also remove explicitly in post_process(). Admittedly, this could certainly use a better docs explanation but the behavior is as-designed.
v
By default why not add the property to the schema then? Hmm I understand the debugging want, but I don't like the idea of adding things that are not from the actual source. A convention like sdc_space_id or something like sounds like the best of both worlds
a
By default why not add the property to the schema then?
Perhaps we should! I'm not sure which approach is better, tbh.
Hmm I understand the debugging want, but I don't like the idea of adding things that are not from the actual source. A convention like sdc_space_id or something like sounds like the best of both worlds
Certainly I think there's space to refine and improve - I don't love an infinite/unbound count of
_sdc_
columns but perhaps a single
_sdc_partition_context
could hold a complex dict of the parent context, and/or something like
_sdc_parent_context
. Would be a much appreciated issue topic and/or office hours topic. 🙂
v
On one extreme you could load all the code needed for the tap into one of those fields
😉