I’m playing with

tap-mysql

and

target-postgres

trying to better understand the

schema extras

. Both are the transferwise versions. I have a json column in the source. I’m trying to override it to make it a varchar column in the target:

schema:
  '*-<table_name>':
    properties:
    type: ['null', 'string']

When I run

meltano invoke --dump=catalog tap-mysql

OR

meltano elt --catlog=catalog.json

doesn’t respect the override OR the catalog:

"properties": {
    "inclusion": "available",
    "type": [
        "null",
        "object"
}

Is it possible that the tap doesn’t respect this? Is there a way to debug that? Thanks!

You might be seeing this? https://github.com/transferwise/pipelinewise-tap-postgres/issues/101

could be. I don’t know that it worked even when I wasn’t running in discovery mode. I’ll do a bit more digging. thanks!

@matt_cooley Did you find out anymore about this? I’m trying to do the same. I have a workaround, but it’s a bit of a hack…

@dean_morin @matt_cooley I'm curios too. I want to overwrite the schema of tap-s3-csv. I'm trying to implement it but.. I got no success yet.

I couldn't get it to work with the extras. I ended up forking the target and overwriting the dictionary it used to assign a type to “object”. Note that the tap I used recognized a json field as an

object

instead of a

string

. Some of them don't. I can point out the lines in the code that made this work for me tomorrow.

I think I found a non-hack fix for this, I’ll link the PR once I have it sorted out

All good! I’ll take a look at these when I have a chance. I’m guessing they can be applied to other taps with “relative” ease anyway. Interestingly enough in my case the tap worked fine (my original comment was a bit off I think). The target however didn’t have the “super” type for redshift, so I rewrote this line to force “object” -> “super”. Obviously I would prefer to have a more permanent fix, but it’s working for me for now.

@dean_morin I have a PR in to fix this issue and someone else does as well

I think I’m the other one! There was the

metadata

issue you were seeing, and I had to do something similar with the

schema

attribute as well