Hey! Getting schema validation errors when trying ...
# troubleshooting
d
Hey! Getting schema validation errors when trying to run
meltano elt <tap> <target>
that are pretty hard to figure out. The error message is claiming that a data type is misspecified but it seems to be mixing up the endpoints.
Copy code
tap-hellobaton   | extractor | time=2021-11-07 14:05:44 name=tap-hellobaton level=INFO message=Beginning full_table sync of 'tasks'...
tap-hellobaton   | extractor | time=2021-11-07 14:05:44 name=tap-hellobaton level=INFO message=Tap has custom mapper. Using 1 provided map(s).
target-snowflake | loader    | CRITICAL '12345.00' is not of type 'integer'
target-snowflake | loader    | CRITICAL 
target-snowflake | loader    | CRITICAL Failed validating 'type' in schema['properties']['annual_contract_value']:
target-snowflake | loader    | CRITICAL     {'type': 'integer'}
target-snowflake | loader    | CRITICAL 
target-snowflake | loader    | CRITICAL On instance['annual_contract_value']:
target-snowflake | loader    | CRITICAL     '12345.00'
tap-hellobaton   | extractor | time=2021-11-07 14:05:44 name=tap-hellobaton level=INFO message=INFO METRIC: {'type': 'timer', 'metric': 'http_request_duration', 'value': 0.741282, 'tags': {'endpoint': '/tasks/', 'http_status_code': 200, 'status': 'succeeded'}}
meltano          | elt       | [Errno 32] Broken pipe
The endpoint
tasks
doesn't have that key listed anywhere in the JSON schema. Weirder still in the json schemas for endpoints that do have the property
annual_contract_value
they are all
type=string
For more context I wrote the tap following the meltano singer spec and am using the meltano variant snowflake target
v
Yeah I wish
Copy code
target-snowflake | loader    | CRITICAL Failed validating 'type' in schema['properties']['annual_contract_value']:
Was a bit more helpful. The issue here is that the tap and the target are not attached.
tap | target
So
annual_contract_value
is coming from a different stream would be my best guess. It just happens that the tap was sending data for a task at the same time. It'd be cool to think about a more generic way to debug these kinds of things easier
https://gitlab.com/meltano/meltano/-/issues/2958 is the best idea I have 🤷
d
Your debug steps on that issue really helped
v
😄 Glad it was helpful, if you have better ideas or additional steps you could add that would be great!