thomas_briggs
03/16/2022, 9:56 PMmappers:
- name: meltano-map-transformer
variant: meltano
pip_url: git+<https://github.com/MeltanoLabs/meltano-map-transform.git>
mappings:
- name: fix-empty-strings
config:
stream_maps:
MyDB-EmptyStringTest:
y: '''foo'' if y == '''' else y'
If it matters, the source is a table in MySQL with two columns: x int and y varchar(10).
When I run meltano run tap-mysql fix-empty-strings target-jsonl
I get the following error:
TypeError: unhashable type: 'list' cmd_type=elb consumer=True name=target-jsonl producer=False stdio=stderr string_id=target-jsonl
I think the error is because the schema being sent to the target looks like this:
{'properties': {'x': {'inclusion': 'automatic', 'minimum': -2147483648, 'maximum': 2147483647, 'type': ['null', 'integer']}, 'y': {'inclusion': 'available', 'maxLength': 10, 'type': [['null', 'string'], 'null']}}, 'type': 'object'}
whereas if I remove the fix-empty-strings transforms the schema sent to the target looks like this:
{'properties': {'x': {'inclusion': 'automatic', 'minimum': -2147483648, 'maximum': 2147483647, 'type': ['null', 'integer']}, 'y': {'inclusion': 'available', 'maxLength': 10, 'type': ['null', 'string']}}, 'type': 'object'}
Any suggestions? Is this a bug in meltano-tap-transformer or is there something wrong with my schema_maps definition? BTW I tried target-csv as well and it has the same problem so I don't think it's an issue with target-jsonl.thomas_briggs
03/17/2022, 6:22 PMy: 'str(''foo'' if y == '''' else y)'
Everything works. In that case CustomStreamMap._eval_type returns StringType instead of the provided default and everything's fine. My Python-fu isn't strong enough to know if that's a bug in _eval_type or _init_functions_and_schema (or something else) though. 😕
@edgar_ramirez_mondragon ,if you'll forgive me for tagging you directly, any insights?