Is there a way through using the `meltano yml` file to ensur Meltano #troubleshooting

Is there a way through using the `meltano.yml` fil...

steven_wang

08/29/2023, 8:38 PM

Is there a way through using the

meltano.yml

file to ensure that catalog types always have a

"null"

option set in them (eg

["string", "null"]

) during the catalog generation process described here? Running into some issues with the shopify tap where the auto-generated catalog specifies a field's type as just

"string"

but the api occasionally returns null. I know a work around would be to manually define the catalog but was hoping there was a more flexible way to do this

steven_wang

08/29/2023, 11:40 PM

realizing that the matatika shopify tap has pre-defined schemas that seem to be out of date. Finding that I need to do a lot of schema overrides on this schema for example: https://github.com/Matatika/tap-shopify/blob/master/tap_shopify/schemas/abandoned_checkout.json

steven_wang

08/29/2023, 11:41 PM

regardless, still curious if there's a way to add

"null"

type to catalogs for all fields in all streams in the meltano.yml config

visch

08/30/2023, 1:17 PM

Check out https://github.com/AutoIDM/shopify2athena/blob/main/meltano.yml I helped some folks with shopify a while back to athena, not sure if this will help or not There's not an easy way to just add null to all the schemas. The easiest fix is to fork the tap and make the changes yourself

Reuben (Matatika)

10/06/2023, 8:31 AM

Hi @steven_wang, we released

v0.2.0

tap-shopify

yesterday, which addresses a load of schema issues like this.

Copy code

meltano lock --update tap-shopify
meltano install --clean extractor tap-shopify

Open in Slack

Previous Next