Hi Everyone, I'm trying to implement a parent-child stream where the parent returns an ID and I then need to use this in the child stream to make a POST request with the ID as part of a list in the request body. I'm using the

prepare_request_payload

method to do this in the child stream. When I run the tap I can see the context element with the ID being passed correctly from the parent stream. However, I cannot get it to pass correctly in the prepare_request_payload of the child stream. I assumed it would be

context["id"]

but when I do this a single ID is passed to all calls of the child stream. The same record is output to the target on each call. The docs here (https://sdk.meltano.com/en/latest/parent_streams.html) only show it being used in the

path

variable wrapped in {} without it being an f string. If I do this

["{id}"]

the context element with the ID from the parent stream updates on each call, which I see in the logs, but the actual API call fails and nothing is output in the target. I believe this is because the element is there in the context but this syntax is incorrect. Is there a step I am missing or something I haven't understood properly?

Using the parent context in

prepare_request_payload

works fine. and

context["id"]

is the correct syntax to use. I'm not sure where it is failing for you. I suppose if you could share a code example of how it is failing for you it might be possible to troubleshoot your code.

Hi @christoph This is the get_child_context method in the parent stream:

def get_child_context(self, record: dict, context: Optional[dict]) -> dict:
        """Return a context dictionary for child streams."""
        return {
            "id": record["waybill_id"],
        }

2022-08-24T09:26:01.333182Z [info     ] time=2022-08-24 10:26:01 name=tap-custom-tap level=INFO message=INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.297746, "tags": {"endpoint": "", "http_status_code": 200, "status": "succeeded", "context": {"id": "1629728091203"}}} cmd_type=elb consumer=False name=tap-custom-tap producer=True stdio=stderr string_id=tap-custom-tap

2022-08-24T09:26:01.333838Z [info     ] time=2022-08-24 10:26:01 name=tap-custom-tap level=INFO message=INFO METRIC: {"type": "counter", "metric": "record_count", "value": 0, "tags": {"stream": "poststream", "context": {"id": "1629728091203"}}} cmd_type=elb consumer=False name=tap-custom-tap producer=True stdio=stderr string_id=tap-custom-tap

2022-08-24T09:26:01.334088Z [info     ] time=2022-08-24 10:26:01 name=tap-custom-tap level=INFO message=Beginning full_table sync of 'poststream' with context: {'id': '1637371467194'}... cmd_type=elb consumer=False name=tap-custom-tapproducer=True stdio=stderr string_id=tap-custom-tap

Can see that the IDs update in the requests on each call.

def prepare_request_payload(
        self, context: Optional[dict], next_page_token: Optional[Any]
    ) -> Optional[dict]:
        """Prepare the data payload for the REST API request."""
        
        return {
            "operation":"QUERY_ID",
            "itemKeys": [context["id"]],
            "pageNo":1,
            "pageSize":10}

When I use the above

[context["id"]]

the same ID is passed to all calls in the context {'id' '123456'} section

The code snippets look all good to me. That's exactly what I'm doing in a few of my own taps as well. The problem must be somewhere else.

Yeah it's going to be something really obvious as well. What's interesting is it's only when I add

context["id"]

in itemKeys that the context id key stays the same throughout. If it's anything else (even just a string such as "id") the context updates

@jazzy Have you defined the

partitions

attribute of the child stream?

@edgar_ramirez_mondragon I have not...

Hmm, then it’s definitely weird. It calls for some debugging to catch the call to

prepare_request_payload

I've been running it with

--log-level debug

but haven't spotted anything that really stands out

sorry I meant using the debugging features of an IDE on your custom tap. That way you can set up breakpoints at key calls (like

prepare_request_payload

). I can try to help with that if you’re not familiar

Ah yes 🤦‍♂️