HI, I'm using meltano to ingest a rest API (an oddly written one) and have discovered that it has an inconsistent document schema. I however want to ingest the field if it exists and set a default value if it does not (say an empty string). tap: https://hub.meltano.com/extractors/tap-rest-api-msdk/ target: https://github.com/crowemi/target-s3 https://sdk.meltano.com/en/latest/stream_maps.html

stream_maps:
  user_email: user_email or ''
  user_gender: user_gender or ''

Around 1-2% of documents do not have one or another of these fields. The issue is that if I don't include a stream maps I get an error on ingestion "json.schema.exceptions.ValidaitonError", 'user_email' is a required property If I do include stream maps, once it actually gets the the document it throws a "singer_sdk.helpers._simpleeval.NameNotDefined" I think this is because in simpleeval

>> simple_eval("user_email or ''", names={"user_email": "<mailto:test@example.com|test@example.com>"})

--> looks like this on good documents, but I'm assuming doesn't get a defined names field on documents where the key doesn't exist Is there any way to have a meltano stream map set a default value if a key doesn't exist but use the existing value if it does?

I think I do something similar in https://github.com/meltano/squared/blob/5411796a958ba770ae2ba23e70b51e5850e8bd26/data/mappers/mappers.meltano.yml#L7 but with a standalone mapper instead. You should be able to do it with stream maps too though. In my case I get either

ipv4prefix

or

ipv6prefix

in the input records but I always want both of them. I renamed them in mine but you should be able to just do

ipv4prefix: record.get('ipv4prefix', '')

or

user_email: record.get('user_email', '')

That looks promising, thanks! I will give that a try.

The following should work:

stream_maps:
  user_email: record.get('user_email', '')
  user_gender: record.get('user_gender', '')