I am using tap shopify variant singer io with target bigquer Meltano #plugins-general

I am using tap-shopify (variant: singer-io, with t...

nil

12/15/2020, 2:10 PM

I am using tap-shopify (variant: singer-io, with target-bigquery) when I only have shopify "customers" selected it works without problem but when I add "orders" the pipeline fails the message in log is:

Copy code

target-bigquery | CRITICAL 'type' or 'anyOf' are required fields in property: {}

Any pointers? (by having only "customer" where it works and adding "order" it fails, so I assume it is problem with tap-shopify and not with target-bigquery)

daniel_pettersson

12/15/2020, 3:18 PM

Hey again, I also ran into issues with target-bigquery being pedantic about the structure. In my case the tap's schema was missing the type field. I fixed it by adding that to the schema json file in the tap

douwe_maan

12/15/2020, 3:41 PM

@nil For some reason tap-shopify doesn't have a type defined for many of the properties in the orders stream as you can see here: https://github.com/singer-io/tap-shopify/blob/master/tap_shopify/schemas/orders.json. You can override the schema in the catalog using https://meltano.com/docs/plugins.html#schema-extra, but it's likely that that won't be picked up correctly until we address https://gitlab.com/meltano/meltano/-/issues/2469.

douwe_maan

12/15/2020, 3:42 PM

In the mean time, I suggest going @daniel_pettersson's route of forking the tap and adding types for each property (based on what you see in the RECORD messages), after which you can contribute those changes back upstream.

nil

12/15/2020, 4:18 PM

@daniel_pettersson @douwe_maan thank you for prompt reply. I Will look into forking tap and changing things. (Also will see if I can contribute to the issue 2469)

nil

12/16/2020, 6:08 PM

Hi, is there a way to find out which part of schema causing problem? (or to print schema and the RECORD to match things?) I have already modified

Copy code

"subtotal_price_set": {},
"total_discounts_set": {},
"total_line_items_price_set": {},
"total_price_set": {},
"total_shipping_price_set": {},
"total_tax_set": {},

 "discounted_price_set": {},
 "price_set": {},

but the error is still there.

douwe_maan

12/16/2020, 7:35 PM

@nil Do you have debug mode enabled (https://meltano.com/docs/command-line-interface.html#debugging)? Then you should see the individual SCHEMA and RECORD messages output by the tap printed with a

tap-shopify (out)

prefix. The SCHEMA message written just before the target complains is the likely culprit 🙂

nil

12/16/2020, 7:37 PM

Cool, will get the debug mode on and see if I can fix the remaining parts 😄

Open in Slack

Previous Next