I'm trying to define a transformation using meltan...
# troubleshooting
t
I'm trying to define a transformation using meltano-map-transformer that replaces empty values in a given field with something else. My mapping config looks like this:
Copy code
mappers:
  - name: meltano-map-transformer
    variant: meltano
    pip_url: git+<https://github.com/MeltanoLabs/meltano-map-transform.git>
    mappings:
    - name: fix-empty-strings
      config:
        stream_maps:
          MyDB-EmptyStringTest:
            y: '''foo'' if y == '''' else y'
If it matters, the source is a table in MySQL with two columns: x int and y varchar(10). When I run
meltano run tap-mysql fix-empty-strings target-jsonl
I get the following error:
Copy code
TypeError: unhashable type: 'list' cmd_type=elb consumer=True name=target-jsonl producer=False stdio=stderr string_id=target-jsonl
I think the error is because the schema being sent to the target looks like this:
Copy code
{'properties': {'x': {'inclusion': 'automatic', 'minimum': -2147483648, 'maximum': 2147483647, 'type': ['null', 'integer']}, 'y': {'inclusion': 'available', 'maxLength': 10, 'type': [['null', 'string'], 'null']}}, 'type': 'object'}
whereas if I remove the fix-empty-strings transforms the schema sent to the target looks like this:
Copy code
{'properties': {'x': {'inclusion': 'automatic', 'minimum': -2147483648, 'maximum': 2147483647, 'type': ['null', 'integer']}, 'y': {'inclusion': 'available', 'maxLength': 10, 'type': ['null', 'string']}}, 'type': 'object'}
Any suggestions? Is this a bug in meltano-tap-transformer or is there something wrong with my schema_maps definition? BTW I tried target-csv as well and it has the same problem so I don't think it's an issue with target-jsonl.
After some poking around in the code I figured out that if I define my transformation as:
y: 'str(''foo'' if y == '''' else y)'
Everything works. In that case CustomStreamMap._eval_type returns StringType instead of the provided default and everything's fine. My Python-fu isn't strong enough to know if that's a bug in _eval_type or _init_functions_and_schema (or something else) though. 😕 @edgar_ramirez_mondragon ,if you'll forgive me for tagging you directly, any insights?