https://linen.dev logo
Join Slack
Powered by
Parent Child streams with a replication key, <http...
# singer-tap-development
v
Parent Child streams with a replication key, https://github.com/meltano/sdk/issues/271 is the last I"m aware of for progress on this. Has anyone made work arounds for this / maybe the SDK supports this now and I missed it? Doing it again right now and I'll probably go with the same work around just trying to see if we have a better way 😄
c
I gave up on trying to make it work. Since the number of partitions and child streams are small-ish in my case. I'm just subclassing one stream for each partition key and have many many child stream classes for now.
v
I just realized I actually didn't need partitions at all, Parent Child streams work with replication keys so no need for me to fret here!
c
Yup. I was wondering about the connection between your question and the partition topic ... I probably don't want to know. 😁
One funny thing with parent child and incremental is the following: If the replication key is the typical "last_modified" timestamp, what happens if the source system makes changes to the records in the child table and updates the last_modified column in the child table, but the associated parent for that child never reflects the modification in its "last_modified" field in the parent table. Meltano simply can't know that it should look at the modified children ... because the associated parents were never modified and hence none of the parents child streams even get triggered for sync in the first place. 🤯
v
I haven't hit a case where I'm using parent / child streams and also have my parent stream be incremental along with my child stream which is the case you're talking about
In my head parent/child streams with replication keys, didn't work. This was wrong which is a good thing for me! Partitions is a seperate concern, super glad this just worked for me I worried about it a bunch and it ended up being a few lines of code
Open in Slack
PreviousNext
Powered by