I am using tap-shopify (variant: singer-io, with t...
# plugins-general
n
I am using tap-shopify (variant: singer-io, with target-bigquery) when I only have shopify "customers" selected it works without problem but when I add "orders" the pipeline fails the message in log is:
Copy code
target-bigquery | CRITICAL 'type' or 'anyOf' are required fields in property: {}
Any pointers? (by having only "customer" where it works and adding "order" it fails, so I assume it is problem with tap-shopify and not with target-bigquery)
d
Hey again, I also ran into issues with target-bigquery being pedantic about the structure. In my case the tap's schema was missing the type field. I fixed it by adding that to the schema json file in the tap
d
@nil For some reason tap-shopify doesn't have a type defined for many of the properties in the orders stream as you can see here: https://github.com/singer-io/tap-shopify/blob/master/tap_shopify/schemas/orders.json. You can override the schema in the catalog using https://meltano.com/docs/plugins.html#schema-extra, but it's likely that that won't be picked up correctly until we address https://gitlab.com/meltano/meltano/-/issues/2469.
In the mean time, I suggest going @daniel_pettersson's route of forking the tap and adding types for each property (based on what you see in the RECORD messages), after which you can contribute those changes back upstream.
n
@daniel_pettersson @douwe_maan thank you for prompt reply. I Will look into forking tap and changing things. (Also will see if I can contribute to the issue 2469)
Hi, is there a way to find out which part of schema causing problem? (or to print schema and the RECORD to match things?) I have already modified
Copy code
"subtotal_price_set": {},
"total_discounts_set": {},
"total_line_items_price_set": {},
"total_price_set": {},
"total_shipping_price_set": {},
"total_tax_set": {},

 "discounted_price_set": {},
 "price_set": {},
but the error is still there.
d
@nil Do you have debug mode enabled (https://meltano.com/docs/command-line-interface.html#debugging)? Then you should see the individual SCHEMA and RECORD messages output by the tap printed with a
tap-shopify (out)
prefix. The SCHEMA message written just before the target complains is the likely culprit 🙂
n
Cool, will get the debug mode on and see if I can fix the remaining parts 😄