Question: is there a way to have a child stream object exit out if not appropriate? In other words, we have one parent stream for accounts. It returns the account id in get_child_context. One child stream will exist for every account, but for another stream the data only exists for some accounts. So in the child stream class for the one that might not exist for an account, can I somehow tell Meltano SDK to not process that child stream? Looking at class RESTStream, it doesn't look like this is currently easy. Override request_records, call super if it exists, but just return otherwise? That might work, but I'll need a change anyway because the headers will change depending on the context, and the current code doesn't pass the context to http_headers. It's also setup as a property, so you can't really pass anything to it. I suppose I can hack it by saving context in self, and overriding the http_headers property, but it should really be a function and not a property, and pass context (from within prepare_request). Can someone look at this?

Hi @fred_reimer! Can you file an issue describing what you're missing https://gitlab.com/meltano/sdk/-/issues/new 🙏? I think we've had other people suggesting we allow passing context to other methods/properties.

An issue may still be helpful so we can improve docs and/or add samples... However, we do have this come up periodically, and there's a solution available if the parent stream's data has enough info to deduce whether the child needs to be called. The method I've used and have seen success with is to pass along the hint within the child context as an additional attribute, and then use

state_partitioning_keys

to exclude that extra field from state tracking logic. In the case of github issues have comments, the issue itself knows the count of comments and if the count of comments is

0

, then there's no need to hit the comments API. Does this help for your use case?

I'm doing this, which works:

def request_records(self, context: Optional[dict]) -> Iterable[dict]:
        """Call super to yield rows only if the security token exists for the AccountId."""
        if str(context["accountId"]) in self.config.get("security_account_tokens"):
            # Save context as authenticator is called by prepare_request without passing context
            self._security_token = self.config["security_account_tokens"][
                str(context["accountId"])
            ]
            for row in super().request_records(context):
                yield row

and I setup an authenticator property that pulls the value:

@property
    def authenticator(self) -> APIKeyAuthenticator:
        """Return a new authenticator object."""
        return APIKeyAuthenticator.create_for_stream(
            self, key="X-API-TOKEN", value=self._security_token, location="header"
        )

Yeah, this looks like it should do the trick! 👍