When loading from my custom extractor to a csv or ...
# plugins-general
j
When loading from my custom extractor to a csv or jsonl target, I'm getting an error that says
Failed validating 'type' in schema['properties']['id']
where
{'type': ['integer']}
And the value is
'4'
. Do I need to convert values before they're sent to stdout?
v
Your tap says that the schema for '4' is an integer while it's actually a string
j
That makes sense. Thank you.
e
Either • the SDK should conform that to an actual int, conform_record_data_types would need to be patched • the schema type for
id
should be string, since that’s what the API sends anyway
j
Thanks Edgar, I knew that the source system's
id
was of type string, but was hoping to do the very basic transformation of converting it to an integer. I know this isn't best practice, so I'll convert my schema back to use string
Thanks for the context about conform_record_data_types
e
@jam a third option is to implement
post_process
in your stream class:
Copy code
class MyStream(...):
   def post_process(self, row, context):
       row["id"] = int(row["id"])
       return row
https://sdk.meltano.com/en/latest/classes/singer_sdk.Stream.html#singer_sdk.Stream.post_process
j
Thank you! I wanted to do some basic post processing to get things setup for the APIs we already have built. I really appreciate this!