Couple questions regarding stream maps settings: I...
# troubleshooting
k
Couple questions regarding stream maps settings: I'm trying to change the key name so that they don't include dashes. I'm using the following set-up
Copy code
some_key: record['some-key']
some-key: __NULL__
This works for string value types, or integer when I use
int(record['some-key'])
but does not work for array types. Is there an example on how to do this with array types? Second issue. When I use
int(record['some-key'])
if the value didn't exist for that field (i.e.
null
) it fails with a message
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
as it probably should. But, I tried to write something where it's still null, if it's null and int parsing if it's not null, but have been unsuccessful. Does anyone have an expression that can be used? Thank you.
v
what happens when you try this with array types? for 2, I don't really understand how expressions work with the mapper. I went and looked at the PR quickly and still don't so 🤷 someday maybe I'll dive and understand simpleeval and this implementation
k
on array types, it raises an error such as
jsonschema.exceptions.ValidationError: ['EMAIL'] is not of type 'string', 'null'
e
If you're using Meltano, you could try overriding the JSON schema inferred by the SDK's stream map logic: https://docs.meltano.com/concepts/plugins/#schema-extra
k
Can I still change the key name using JSON schema? I want to keep the exact nested structure that the column had, but just need the key name to be changed.