hey i m having some trouble using a custom catalog with the Meltano #troubleshooting

hey, i’m having some trouble using a custom catalo...

prratek_ramchandani

06/02/2021, 1:11 PM

hey, i’m having some trouble using a custom catalog with the default variant of

tap-postgres

. i’m pointing it to my custom catalog file which has a modified schema for most streams and see this in the logs

Copy code

meltano         | Found catalog in /path/to/meltano/extract/tap-postgres.catalog.json

but the tap still outputs the default schema. i noticed that the tap takes a

--properties

option and not

--catalog

. could that be responsible for the issue here?

prratek_ramchandani

06/02/2021, 1:23 PM

i also found this in Slack, which seems to suggest the tap might just not be using the provided schema? i tried overriding the

schema

in meltano.yml too https://meltano.slack.com/archives/C01TCRBBJD7/p1621971324081500?thread_ts=1621970218.073800&cid=C01TCRBBJD7

taylor

06/02/2021, 1:32 PM

Meltano should be able to do the right thing if the

properties

capability is defined. see https://meltano.slack.com/archives/C01TCRBBJD7/p1621526498044200?thread_ts=1621519111.034600&cid=C01TCRBBJD7

prratek_ramchandani

06/02/2021, 1:56 PM

huh okay - do you think the tap is just not using the override schema then? any ideas on a workaround? the tap is identifying some fields as being of type

["null", "object", "array"]

and bigquery doesn’t know what to do with that. the target throws this when it gets to one of those fields because the

object

type field doesn’t declare `properties`:

Copy code

"TypeError: argument of type 'NoneType' is not iterable"

douwe_maan

06/02/2021, 2:11 PM

do you think the tap is just not using the override schema then?

Yeah, this is likely. We have an issue about addressing this in Meltano: https://gitlab.com/meltano/meltano/-/issues/2469. Another approach would use https://gitlab.com/meltano/meltano/-/issues/2300, which isn't implemented yet either. The only real workaround for now would be to patch the tap to respect the overridden schema.

douwe_maan

06/02/2021, 2:11 PM

I wonder why the tap is claiming that the field can be object or array. Is this a json(b) column?

prratek_ramchandani

06/02/2021, 2:41 PM

yeah it is. i happen to know it’ll never be an array but to be fair the tap wouldn’t know that.

prratek_ramchandani

06/02/2021, 2:44 PM

i see the June 11 milestone on that issue - if it might actually get implemented by mid-june i can just wait but if you think it might take longer then i’ll take a stab at foking the tap and messing around

douwe_maan

06/02/2021, 2:56 PM

It's likely to take longer, there is some refactoring work that needs to happen first. We may start on it then, but I wouldn't expect it to be released by mid-June

prratek_ramchandani

06/02/2021, 2:56 PM

got it, thanks!

prratek_ramchandani

06/02/2021, 3:13 PM

i’m trying to find an example of a tap checking to see if someone has specified a custom catalog or schema and then using it. would you be able to point me to where the singer SDK looks for a custom catalog? everything i see seems to take the catalog as an argument haha

prratek_ramchandani

06/02/2021, 3:20 PM

nvm i think i found it https://gitlab.com/meltano/singer-sdk/-/blob/main/singer_sdk/tap_base.py#L252

prratek_ramchandani

06/02/2021, 3:40 PM

okay for anyone interested, this function call in the tap appears to do a new discovery during the sync and replace the existing catalog with the new one. not sure if there’s some other purpose this was serving but commenting out gets me a different error that i know is my fault, which is great!

Open in Slack

Previous Next