We are currently experiencing an issue with ‘meltano-map-transformer’ when a stream has a parent stream. We’ve developed a custom tap to interact with an API. For this API, to get the details, we have to first get a list of parent objects (in this case help desk ticket numbers), and then call a separate API to get the details of each ticket. The first ticket uses the alias defined in the config correctly, but all subsequent calls do not use the alias. We have done some troubleshooting and have found the following. Within the current Singer SDK version, the PluginMapper class in mapper.py has this function 'register_raw_stream_schema' where the code is 'popping' values from stream definition. I think the intention was the for loop here would copy the stream definition (stream_def), but that is not the case.

for stream_map_key, stream_def in self.stream_maps_dict.items():
   stream_alias: str = stream_map_key
   source_stream: str = stream_map_key

Within the for loop, the following is later executed, and it is altering the stream definition:

if MAPPER_SOURCE_OPTION in stream_def:
   source_stream = stream_def.pop(MAPPER_SOURCE_OPTION)

if source_stream != stream_name:
   # Not a match
   continue

if MAPPER_ALIAS_OPTION in stream_def:
   stream_alias = stream_def.pop(MAPPER_ALIAS_OPTION)

When I altered the for loop locally to make a copy of the stream definition, the alias configuration was maintained and we got the expected outcome.

for stream_map_key, y in self.stream_maps_dict.items():
   stream_def = dict(y)  
   stream_alias: str = stream_map_key
   source_stream: str = stream_map_key

Would anyone else agree that the intent was to copy the stream definition? Any other thoughts or questions, please let me know. Thank you!

Oh, we gotta be more careful with mutable objects I guess 😅. Thanks for reporting @john_rhodes! So, I think because of the partitions, the schema message for the child streams is received multiple times, but the stream config is mutated the first time. That’s a bug! Issue logged: https://github.com/meltano/sdk/issues/1521 PRs welcome since I might not be able to jump to it this week.

Thanks, @john_rhodes for reporting and @edgar_ramirez_mondragon for logging. Agreed this definitely looks like a bug. The analysis is very helpful! 🙏