Hey all, I'm trying to use stream maps to fabricate a PK when a tap doesn't emit one. The mapper is detecting the key properties override, but then not updating the schema message to the target. Any ideas? My

meltano.yml

looks like:

config:
      stream_maps:
        events:
          id: "md5(customer_id + edit_id + date + event)"
          __key_properties__: ["id"]

and some log output:

tap-mytap               | extractor | time=2021-11-26 09:22:42 name=root level=INFO message=Found stream map override for 'events' key properties: ['id']
tap-mytap               | extractor | time=2021-11-26 09:22:42 name=tap-mytap level=INFO message=Beginning incremental sync of 'events'...
tap-mytap               | extractor | time=2021-11-26 09:22:42 name=tap-mytap level=INFO message=Tap has custom mapper. Using 1 provided map(s).
tap-mytap (out)         | extractor | {"type": "SCHEMA", "stream": "events", "schema": {"properties": {"stylist_id": {"type": ["string", "null"]}, "customer_id": {"type": ["string", "null"]}, "edit_id": {"type": ["string", "null"]}, "edit_title": {"type": ["string", "null"]}, "date": {"format": "date-time", "type": ["string", "null"]}, "event": {"type": ["string", "null"]}, "product_id": {"type": ["string", "null"]}, "external_id": {"type": ["string", "null"]}, "product_url": {"type": ["string", "null"]}, "customer_email": {"type": ["string", "null"]}, "customer_first_name": {"type": ["string", "null"]}, "customer_last_name": {"type": ["string", "null"]}, "customer_phone_number": {"type": ["string", "null"]}, "id": {"type": ["string", "null"]}}, "type": "object"}, "key_properties": [], "bookmark_properties": ["date"]}

Not sure what is going on here 🤔 My only thought is that the

__key_properties__

key is not being correctly parsed from YAML. Have you tried wrapping it in double quotes?

I wondered that, but the log line generated by https://gitlab.com/meltano/sdk/-/blob/main/singer_sdk/mapper.py#L308 would seem to confirm that it's being read?

Me neither. I think this is a bug 🐛 They are set/saved here but never applied to the returned

transformed_schema

🤔 Have scanned the SDK tests and can't find one that covers

__key_properties__

either, so even more likely this is a bug 😅 Do you mind creating an issue? I'm not set up to easily work on the SDK yet, but @edgar_ramirez_mondragon should be on later today and has worked a lot on the SDK 👍

@ken_payne @niall_woodward This indeed looks like a bug! The transformed PK is never applied: https://gitlab.com/meltano/sdk/-/blob/main/singer_sdk/streams/core.py#L691. Filing an issue...

Thanks Edgar 🙏 I see you have already raised a MR too - that was quick 😅🙌🚀

Thanks @edgar_ramirez_mondragon!