Hi everyone, we use tap-mysql and target-athena. T...
# troubleshooting
a
Hi everyone, we use tap-mysql and target-athena. Tap/Target works very well on some entities. But, an issue is raising on one entity with the following exception
jsonschema.exceptions.ValidationError: 48.867082 is not a multiple of 1e-06.
This issue seems to appear during the extractor step. In addition the previous error, the extractor raise error as below:
Copy code
Failed validating 'multipleOf' in schema['properties']['lat']: 
2022-03-08T18:31:59.748597Z [info     ]     {'inclusion': 'available', 
2022-03-08T18:31:59.748733Z [info     ]      'multipleOf': 1e-06,      
2022-03-08T18:31:59.748840Z [info     ]      'type': ['null', 'number']} 
2022-03-08T18:31:59.748960Z [info     ]                                
2022-03-08T18:31:59.831977Z [info     ] On instance['lat']:            
2022-03-08T18:31:59.833501Z [info     ]     48.867082
We try to resolve it with schema setting to override the catalog schema from tap-mysql.
Copy code
meltano config tap-mysql set _schema  some_stream_id lat multipleOf 1e-08
meltano config tap-mysql set _metadata some_stream_id lat multipleOf 1e-08
But, any previous command resolve the issue. Have you ever encountered this problem? This issue seems already appear on other target like target-csv (https://gitlab.com/meltano/target-csv/-/issues/3)
Bump the sdk to v0.4.4 not resolve the issue
We tried to add
schema = utils.float_to_decimal(self.schema)
inline 130 of sinks.py
like target-postgres but this fix always return the same issue about multipleOf
e
I think this might be solved by loading numbers as python `decimal.Decimal`:
Copy code
>>> import jsonschema
>>> jsonschema.validate(48.867082, {"type": "number", "multipleOf": 0.000001})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/edgarramirez/Library/Caches/pypoetry/virtualenvs/singer-sdk-S0flctL8-py3.9/lib/python3.9/site-packages/jsonschema/validators.py", line 934, in validate
    raise error
jsonschema.exceptions.ValidationError: 48.867082 is not a multiple of 1e-06

Failed validating 'multipleOf' in schema:
    {'multipleOf': 1e-06, 'type': 'number'}

On instance:
    48.867082
>>> import decimal
>>> jsonschema.validate(decimal.Decimal("48.867082"), {"type": "number", "multipleOf": decimal.Decimal("0.000001")})
I think we can do that in the SDK so
MeltanoLabs/target-athena
and
MeltanoLabs/target-csv
would inherit that behavior. I popped https://gitlab.com/meltano/sdk/-/issues/346 to discuss the solution. @alexandre_brehelin can you leave a 👍 if you think that'll solve your issue or a comment to further explain the problem you're having?
a
Hi Edgar, I think your suggestion is good.
Currently, I managed to solve the issue with some transformation on sinks.py file from the target-athena repo.
Override _validate_and_parse function like
self._validator.validate(utils.float_to_decimal(record))
and
schema = utils.float_to_decimal(self.schema)
in process_batch
but provide the fix in the sdk is a better solution like your suggest. Like the others target can benefit from it.