So I’ve been messing around with parent-child streams, and I’d like to mention couple things I’ve found. •

get_child_context()

gets called after

post_process()

. Would it not make more sense to call

get_child_context()

on the unprocessed record? For example, I had to output a json string as the value of a column in the final record, but I needed a value from within the json string for

get_child_context()

.

get_child_context()

being called on the post-processed record caused me to have to

json.loads(record)

, which seems like an unnecessary step. • It would be nice to be able to access the

context

passed to a child stream from any section of the stream. For example, my parent stream might pass a list of ids to query in the child, but the child endpoint only takes a single id per query. It would be helpful to be able to access the

context

from

get_next_page_token()

and pop the id off as needed instead of writing indexing logic within

get_next_page_token()

. I’m sure there are other areas that

context

would be helpful to access that aren’t currently accessible, but this is just the most immediate example I have. • It could also be helpful to have some sort of mechanism to build up a

context

in a parent stream to pass to a child. Since

get_child_context()

is called on each record, you may be able to only pull on value from that record, but it’s possible that the child stream can take in a list of values per request. So if there are N records produced from the parent stream, the current implementation means you have to make N calls to the API via the child stream. Building a context to pass to the child would mean you could make just one call. Maybe there is a way to access the

context

from anywhere within a stream, but I’m just unaware of it.

@michael_cooper - I like both of these suggestions and I think they make a lot of sense. 👍 Both would be breaking changes, unfortunately, but we can plan them for an update where similar refactoring might be happening anyway. Certainly would like to have both of these points solid in our 1.0, when we get there. Do you mind logging a quick issue for each?

Yeah, that would be good.