Using

tap-rest-api-msdk

is it possible to infer all types from the results of an api call and explicitily define the type of one of the fields? The field

mentions_disabled

has almost all values equals to

null

but one of them

false

. Schema was inferred with

null

and this error ocurrs when it tries to read the

false

value.

Failed validating 'type' in schema['properties']['mentions_disabled']

jsonschema.exceptions.ValidationError: False is not of type 'null'

I have already tried to configure

num_inference_keys

but the same error ocurrs.

I think it's unfortunately all-or-nothing: either infer the type of all fields, or pass a custom JSON schema for the whole record with

schema

. Do log an issue on their repo if you can. The maintainers are fairly responsive: https://github.com/Widen/tap-rest-api-msdk/issues.

The challenge is that it is sampling a number of records. You would need to read every record first to accurately detect the types. If you were doing an incremental ingestion, the next batch of records maybe null so there is no hint for the schema. With an API like that I recommend you pass a custom schema as @edgar_ramirez_mondragon mentioned or defer the flattening and discovery to the target.