Parent and Child Stream contexts are shared between any Chil Meltano #singer-tap-development

Parent and Child Stream contexts are shared betwee...

visch

04/10/2025, 8:21 PM

Parent and Child Stream contexts are shared between any Child Streams that have a shared Parent. Noticed an SDK thing today that I don't think used to be the case but this is a weird one. Would have to put together a tap to show this 🧵

✅ 1

visch

04/10/2025, 8:21 PM

Parent -> Child Stream 1 If Child Stream 1 edits the context provided to it from its parent, Child Stream 2 also gets all of those context's sent to it Parent -> Child Stream 2 https://github.com/visch/tap-sharedparentcontext is an example of it

Copy code

visch@visch-work-2:~/git/tap-sharedparentcontext$ meltano invoke tap-sharedparentcontext > out

2025-04-10 16:19:52,171 | INFO     | tap-sharedparentcontext | Added 'users_child_one' as child stream to 'users'
2025-04-10 16:19:52,171 | INFO     | tap-sharedparentcontext | Added 'users_child_two' as child stream to 'users'
2025-04-10 16:19:52,171 | INFO     | tap-sharedparentcontext.users | Beginning full_table sync of 'users'...
2025-04-10 16:19:52,171 | INFO     | tap-sharedparentcontext.users | Tap has custom mapper. Using 1 provided map(s).
2025-04-10 16:19:52,172 | INFO     | tap-sharedparentcontext.users_child_one | Beginning incremental sync of 'users_child_one' with context: {'id': '1'}...
2025-04-10 16:19:52,172 | INFO     | tap-sharedparentcontext.users_child_one | Tap has custom mapper. Using 1 provided map(s).
2025-04-10 16:19:52,172 | INFO     | tap-sharedparentcontext.users_child_one | Starting incremental sync with bookmark value: 2010-01-01T00:00:00Z
2025-04-10 16:19:52,172 | INFO     | tap-sharedparentcontext.users_child_one | Context User Child One: {'id': '1', 'url_parameter_for_user_child_one': '1'}
2025-04-10 16:19:52,172 | WARNING  | tap-sharedparentcontext.users_child_one | Properties ('url_parameter_for_user_child_one',) were present in the 'users_child_one' stream but not found in catalog schema. Ignoring.
2025-04-10 16:19:52,172 | WARNING  | singer_sdk           | Stream is assumed to be unsorted, progress is not resumable if interrupted
2025-04-10 16:19:52,172 | INFO     | singer_sdk.metrics   | METRIC: {"type": "timer", "metric": "sync_duration", "value": 0.0002925395965576172, "tags": {"stream": "users_child_one", "pid": 152722, "context": {"id": "1", "url_parameter_for_user_child_one": "1"}, "status": "succeeded"}}
2025-04-10 16:19:52,172 | INFO     | singer_sdk.metrics   | METRIC: {"type": "counter", "metric": "record_count", "value": 1, "tags": {"stream": "users_child_one", "pid": 152722, "context": {"id": "1", "url_parameter_for_user_child_one": "1"}}}
2025-04-10 16:19:52,172 | INFO     | tap-sharedparentcontext.users_child_two | Beginning incremental sync of 'users_child_two' with context: {'id': '1', 'url_parameter_for_user_child_one': '1'}...
2025-04-10 16:19:52,172 | INFO     | tap-sharedparentcontext.users_child_two | Tap has custom mapper. Using 1 provided map(s).
2025-04-10 16:19:52,172 | INFO     | tap-sharedparentcontext.users_child_two | Starting incremental sync with bookmark value: 2010-01-01T00:00:00Z
2025-04-10 16:19:52,172 | INFO     | tap-sharedparentcontext.users_child_two | Context User Child Two: {'id': '1', 'url_parameter_for_user_child_one': '1'}
2025-04-10 16:19:52,173 | WARNING  | tap-sharedparentcontext.users_child_two | Properties ('url_parameter_for_user_child_one',) were present in the 'users_child_two' stream but not found in catalog schema. Ignoring.
2025-04-10 16:19:52,173 | WARNING  | singer_sdk           | Stream is assumed to be unsorted, progress is not resumable if interrupted
2025-04-10 16:19:52,173 | INFO     | singer_sdk.metrics   | METRIC: {"type": "timer", "metric": "sync_duration", "value": 0.00020647048950195312, "tags": {"stream": "users_child_two", "pid": 152722, "context": {"id": "1", "url_parameter_for_user_child_one": "1"}, "status": "succeeded"}}
2025-04-10 16:19:52,173 | INFO     | singer_sdk.metrics   | METRIC: {"type": "counter", "metric": "record_count", "value": 1, "tags": {"stream": "users_child_two", "pid": 152722, "context": {"id": "1", "url_parameter_for_user_child_one": "1"}}}
2025-04-10 16:19:52,173 | INFO     | singer_sdk.metrics   | METRIC: {"type": "timer", "metric": "sync_duration", "value": 0.0014579296112060547, "tags": {"stream": "users", "pid": 152722, "context": {}, "status": "succeeded"}}
2025-04-10 16:19:52,173 | INFO     | singer_sdk.metrics   | METRIC: {"type": "counter", "metric": "record_count", "value": 1, "tags": {"stream": "users", "pid": 152722, "context": {}}}

Out

Copy code

{"type":"SCHEMA","stream":"users","schema":{"properties":{"name":{"type":["string","null"]},"id":{"description":"The user's system ID","type":["string","null"]},"age":{"description":"The user's age in years","type":["integer","null"]},"email":{"description":"The user's email address","type":["string","null"]},"street":{"type":["string","null"]},"city":{"type":["string","null"]},"state":{"description":"State name in ISO 3166-2 format","type":["string","null"]},"zip":{"type":["string","null"]}},"type":"object","$schema":"<https://json-schema.org/draft/2020-12/schema>"},"key_properties":["id"]}
{"type":"SCHEMA","stream":"users_child_one","schema":{"properties":{"name":{"type":["string","null"]},"id":{"type":["string","null"]},"modified":{"format":"date-time","type":["string","null"]}},"type":"object","$schema":"<https://json-schema.org/draft/2020-12/schema>"},"key_properties":["id"],"bookmark_properties":["modified"]}
{"type":"RECORD","stream":"users_child_one","record":{"id":"1","name":"John Doe","modified":"2021-01-01T00:00:00Z"},"time_extracted":"2025-04-10T20:19:52.172408+00:00"}
{"type":"STATE","value":{"bookmarks":{"users":{"starting_replication_value":null},"users_child_one":{"partitions":[{"context":{"id":"1","url_parameter_for_user_child_one":"1"},"replication_key":"modified","replication_key_value":"2021-01-01T00:00:00Z"}]}}}}
{"type":"SCHEMA","stream":"users_child_two","schema":{"properties":{"name":{"type":["string","null"]},"id":{"type":["string","null"]},"modified":{"format":"date-time","type":["string","null"]}},"type":"object","$schema":"<https://json-schema.org/draft/2020-12/schema>"},"key_properties":["id"],"bookmark_properties":["modified"]}
{"type":"RECORD","stream":"users_child_two","record":{"id":"1","name":"John Doe","modified":"2021-01-01T00:00:00Z"},"time_extracted":"2025-04-10T20:19:52.173075+00:00"}
{"type":"STATE","value":{"bookmarks":{"users":{"starting_replication_value":null},"users_child_one":{"partitions":[{"context":{"id":"1","url_parameter_for_user_child_one":"1"},"replication_key":"modified","replication_key_value":"2021-01-01T00:00:00Z"}]},"users_child_two":{"partitions":[{"context":{"id":"1","url_parameter_for_user_child_one":"1"},"replication_key":"modified","replication_key_value":"2021-01-01T00:00:00Z"}]}}}}
{"type":"RECORD","stream":"users","record":{"id":"1","name":"John Doe","age":30},"time_extracted":"2025-04-10T20:19:52.173446+00:00"}
{"type":"STATE","value":{"bookmarks":{"users":{},"users_child_one":{"partitions":[{"context":{"id":"1","url_parameter_for_user_child_one":"1"},"replication_key":"modified","replication_key_value":"2021-01-01T00:00:00Z"}]},"users_child_two":{"partitions":[{"context":{"id":"1","url_parameter_for_user_child_one":"1"},"replication_key":"modified","replication_key_value":"2021-01-01T00:00:00Z"}]}}}}
{"type":"STATE","value":{"bookmarks":{"users":{},"users_child_one":{"partitions":[{"context":{"id":"1","url_parameter_for_user_child_one":"1"},"replication_key":"modified","replication_key_value":"2021-01-01T00:00:00Z"}]},"users_child_two":{"partitions":[{"context":{"id":"1","url_parameter_for_user_child_one":"1"},"replication_key":"modified","replication_key_value":"2021-01-01T00:00:00Z"}]}}}}
{"type":"STATE","value":{"bookmarks":{"users":{},"users_child_one":{"partitions":[{"context":{"id":"1","url_parameter_for_user_child_one":"1"},"replication_key":"modified","replication_key_value":"2021-01-01T00:00:00Z"}]},"users_child_two":{"partitions":[{"context":{"id":"1","url_parameter_for_user_child_one":"1"},"replication_key":"modified","replication_key_value":"2021-01-01T00:00:00Z"}]}}}}

visch

04/10/2025, 8:21 PM

That repo you can clone and just run

meltano invoke tap-sharedparentcontext

there's no config or anything required

visch

04/10/2025, 8:22 PM

The unexpected behavior is here https://github.com/visch/tap-sharedparentcontext/blob/af3a72e9f0dc2c61aeb977ba77ecadfab1429b20/tap_sharedparentcontext/streams.py#L88 and https://github.com/visch/tap-sharedparentcontext/blob/af3a72e9f0dc2c61aeb977ba77ecadfab1429b20/tap_sharedparentcontext/streams.py#L119

UsersChildTwoStream

gets context changes from

UsersChildOneStream

visch

04/10/2025, 8:27 PM

Copy code

2025-04-10 16:19:52,172 | INFO     | tap-sharedparentcontext.users_child_two | Context User Child Two: {'id': '1', 'url_parameter_for_user_child_one': '1'}

Shows that the user_child_two stream has context from user_child_one

visch

04/10/2025, 8:39 PM

https://sdk.meltano.com/en/v0.45.5/context_object.html#best-practices-for-using-context We shouldn't modify context just as a note to everyone! I was doing it for a specific reason here and thought I knew the consequences but missed this

👀 1

Edgar Ramírez (Arch.dev)

04/10/2025, 11:33 PM

Hey Derek! What's the use case for mutating the context in a child stream? You correctly pointed out to the docs discouraging that, but I'm curious if there's something missing in the SDK to help your use case.

visch

04/11/2025, 10:05 AM

Copy code

def get_records(self, context: Context | None) -> t.Iterable[dict[str, t.Any]]:
        """Return a generator of record-type dictionary objects.

        Each record emitted should be a dictionary of property names to their values.

        Args:
            context: Stream partition or context dictionary.

        Yields:
            One item per (possibly processed) record in the API.
        """
        context = copy.deepcopy(context)
        # Context object is mutable and can affect other Streams
        for partition in self.inventory_partitions():
            context["inventory_type"] = partition["inventory_type"]
            context["inventory_source_type"] = partition["inventory_source_type"]
            for record in self.request_records(context):
                transformed_record = self.post_process(record, context)
                if transformed_record is None:
                    # Record filtered out during post_process()
                    continue
                yield transformed_record

@Edgar Ramírez (Arch.dev) I tried using

partitions

but this is a Parent Child Stream where I need some data from the parent, and swapping the context is nice as my request payload changes based on the data in

inventory_partitions

I've written something to make parent/child partitions work in this scenario when I needed state but I don't need state for this stream so this worked for me. Maybe I'm missing something and can just use partitions?

Edgar Ramírez (Arch.dev)

04/11/2025, 3:56 PM

Maybe it's back to this old issue? https://github.com/meltano/sdk/issues/271

ty 1

visch

04/11/2025, 5:36 PM

Yep 🙂 Wrongly thought editing context would be ok, does make me think I should dive context a bit more and then fix this in the SDK but it's one of those things that happens is a bunch of pain once and then isn't pain again for a long time!

8 Views

Open in Slack

Previous Next