https://linen.dev logo
Join Slack
Powered by
I've been experimenting with the Shopify Tap from ...
# troubleshooting
b
I've been experimenting with the Shopify Tap from the Matatika team and having some issues getting the Tap to run to extract products from a store. I believe I am getting authentication back, but the updating of incremental state fails due to a key error. From some prior conversations in the Slack I am guessing this could be solvable with some secret config changes? My config is reasonably basic and I see the error when invoking the tap directly without a target.
Copy code
extractors:
  - name: tap-shopify
    variant: matatika
    config:
      store: {SECRET}
      access_token: {SECRET}
      is_plus_account: True
      start_date: 2023-01-01T00:00:00Z
    select:
    - products
Error in 🧵 , would be great if anyone has seen this before
```time=2023-07-31 170022 name=tap-shopify level=INFO message=tap-shopify v0.0.1, Meltano SDK v0.4.9) time=2023-07-31 170022 name=tap-shopify level=INFO message=Skipping parse of env var settings... time=2023-07-31 170022 name=tap-shopify level=INFO message=Config validation passed with 0 warnings. time=2023-07-31 170022 name=root level=INFO message=Operator '__else__=None' was not found. Unmapped streams will be included in output. time=2023-07-31 170022 name=tap-shopify level=INFO message=Added 'inventory_items' as child stream to 'inventory_levels' time=2023-07-31 170022 name=tap-shopify level=INFO message=Added 'inventory_levels' as child stream to 'locations' time=2023-07-31 170022 name=tap-shopify level=INFO message=Added 'transactions' as child stream to 'orders' time=2023-07-31 170022 name=tap-shopify level=INFO message=Skipping deselected stream 'abandoned_checkouts'. time=2023-07-31 170022 name=tap-shopify level=INFO message=Skipping deselected stream 'collects'. time=2023-07-31 170022 name=tap-shopify level=INFO message=Skipping deselected stream 'custom_collections'. time=2023-07-31 170022 name=tap-shopify level=INFO message=Skipping deselected stream 'customers'. time=2023-07-31 170022 name=tap-shopify level=INFO message=Skipping deselected stream 'inventory_items'. time=2023-07-31 170022 name=tap-shopify level=INFO message=Skipping deselected stream 'inventory_levels'. time=2023-07-31 170022 name=tap-shopify level=INFO message=Skipping deselected stream 'locations'. time=2023-07-31 170022 name=tap-shopify level=INFO message=Skipping deselected stream 'metafields'. time=2023-07-31 170022 name=tap-shopify level=INFO message=Skipping deselected stream 'orders'. time=2023-07-31 170022 name=tap-shopify level=INFO message=Beginning incremental sync of 'products'... time=2023-07-31 170022 name=tap-shopify level=INFO message=Tap has custom mapper. Using 1 provided map(s). {"type": "SCHEMA", "stream": "products", "schema": {"properties": {"id": {"type": "integer"}}, "type": "object"}, "key_properties": ["id"], "bookmark_properties": ["updated_at"]} time=2023-07-31 170022 name=tap-shopify level=INFO message=INFO METRIC: {'type': 'timer', 'metric': 'http_request_duration', 'value': 0.199651, 'tags': {'endpoint': '/api/2022-01/products.json', 'http_status_code': 200, 'status': 'succeeded'}} {"type": "RECORD", "stream": "products", "record": {"id": 6885950652493}, "time_extracted": "2023-07-31T210022.331551Z"} Traceback (most recent call last): File "/Users/benjamintheunissen/Dev/Shaped/shaped-meltano/.meltano/extractors/tap-shopify/venv/bin/tap-shopify", line 8, in <module> sys.exit(Tap_Shopify.cli()) File "/Users/benjamintheunissen/Dev/Shaped/shaped-meltano/.meltano/extractors/tap-shopify/venv/lib/python3.10/site-packages/click/core.py", line 1157, in call return self.main(*args, **kwargs) File "/Users/benjamintheunissen/Dev/Shaped/shaped-meltano/.meltano/extractors/tap-shopify/venv/lib/python3.10/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/Users/benjamintheunissen/Dev/Shaped/shaped-meltano/.meltano/extractors/tap-shopify/venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/benjamintheunissen/Dev/Shaped/shaped-meltano/.meltano/extractors/tap-shopify/venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) File "/Users/benjamintheunissen/Dev/Shaped/shaped-meltano/.meltano/extractors/tap-shopify/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 499, in cli tap.sync_all() File "/Users/benjamintheunissen/Dev/Shaped/shaped-meltano/.meltano/extractors/tap-shopify/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 379, in sync_all stream.sync() File "/Users/benjamintheunissen/Dev/Shaped/shaped-meltano/.meltano/extractors/tap-shopify/venv/lib/python3.10/si…
r
Hi @ben_theunissen, sorry you're getting an error here. We do test this tap every day, but not incrementally at the moment (perhaps we should look into do doing this i.e. full sync once per week, incremental sync daily @daniel_walker @gabriel_ciobotaru). I'm not 100% sure if there is a workaround you can do here in the meantime (I'll let others answer that), but I can see that the tap is using quite and old version of the SDK. It's possible that bumping this to a more recent version solves your issue, especially since I don't see any 
tap-shopify
code calls in the error traceback. I'll look into this in the morning and send any updates here. 🙂
Of course, since I don't have access to your data/specific use-case, I can't guarantee that the dependency upgrade will work for you (as explained here also). I'd ask if you would be able to verify the fix from your end if the issue was more specific, but given that the SDK dependency version is as old as it is, we'll probably end up doing the upgrade regardless since I see it as a net-positive.
b
Thanks very much for looking into this, I’ll give it a look first thing tomorrow!
r
@ben_theunissen I've finished with the changes just now. To upgrade, you should just need to
Copy code
meltano lock --update tap-shopify
and then
Copy code
meltano install --clean extractor tap-shopify
b
Fantastic, just got it updated and working then! At the risk of sounding inexperienced with nested Meltano streams, but as I invoke the tap to get 
orders
, I am seeing a list of records containing:
Copy code
{"type": "RECORD", "stream": "products", "record": {"id": 6829097254989, "updated_at": "2023-08-01T08:49:58-07:00"}, "time_extracted": "2023-08-01T16:19:45.705832+00:00"}
{"type": "RECORD", "stream": "products", "record": {"id": 6874459504717, "updated_at": "2023-08-01T08:59:20-07:00"}, "time_extracted": "2023-08-01T16:19:45.705891+00:00"}
Without the other required fields provided on the schema that is discovered, do I need some additional config here or could the API have changed under the hood?
```2023-08-01T162227.657124Z [info ] jsonschema.exceptions.ValidationError: 'admin_graphql_api_id' is a required property cmd_type=loader name=target-parquet run_id=66f5f6a0-2e97-4673-94e5-53e67bd6b233 state_id=2023-08-01T162225--tap-shopify--target-parquet stdio=stderr 2023-08-01T162227.657174Z [info ] cmd_type=loader name=target-parquet run_id=66f5f6a0-2e97-4673-94e5-53e67bd6b233 state_id=2023-08-01T162225--tap-shopify--target-parquet stdio=stderr 2023-08-01T162227.657221Z [info ] Failed validating 'required' in schema: cmd_type=loader name=target-parquet run_id=66f5f6a0-2e97-4673-94e5-53e67bd6b233 state_id=2023-08-01T162225--tap-shopify--target-parquet stdio=stderr 2023-08-01T162227.657269Z [info ] {'properties': {'id': {'type': 'integer'}, cmd_type=loader name=target-parquet run_id=66f5f6a0-2e97-4673-94e5-53e67bd6b233 state_id=2023-08-01T162225--tap-shopify--target-parquet stdio=stderr 2023-08-01T162227.657316Z [info ] 'updated_at': {'format': 'date-time', cmd_type=loader name=target-parquet run_id=66f5f6a0-2e97-4673-94e5-53e67bd6b233 state_id=2023-08-01T162225--tap-shopify--target-parquet stdio=stderr 2023-08-01T162227.657372Z [info ] 'type': ['null', 'string']}}, cmd_type=loader name=target-parquet run_id=66f5f6a0-2e97-4673-94e5-53e67bd6b233 state_id=2023-08-01T162225--tap-shopify--target-parquet stdio=stderr 2023-08-01T162227.657419Z [info ] 'required': ['admin_graphql_api_id', cmd_type=loader name=target-parquet run_id=66f5f6a0-2e97-4673-94e5-53e67bd6b233 state_id=2023-08-01T162225--tap-shopify--target-parquet stdio=stderr 2023-08-01T162227.657467Z [info ] 'body_html', cmd_type=loader name=target-parquet run_id=66f5f6a0-2e97-4673-94e5-53e67bd6b233 state_id=2023-08-01T162225--tap-shopify--target-parquet stdio=stderr 2023-08-01T162227.657513Z [info ] 'created_at', cmd_type=loader name=target-parquet run_id=66f5f6a0-2e97-4673-94e5-53e67bd6b233 state_id=2023-08-01T162225--tap-shopify--target-parquet stdio=stderr 2023-08-01T162227.657559Z [info ] 'handle', cmd_type=loader name=target-parquet run_id=66f5f6a0-2e97-4673-94e5-53e67bd6b233 state_id=2023-08-01T162225--tap-shopify--target-parquet stdio=stderr 2023-08-01T162227.657605Z [info ] 'id', cmd_type=loader name=target-parquet run_id=66f5f6a0-2e97-4673-94e5-53e67bd6b233 state_id=2023-08-01T162225--tap-shopify--target-parquet stdio=stderr 2023-08-01T162227.657651Z [info ] 'image', cmd_type=loader name=target-parquet run_id=66f5f6a0-2e97-4673-94e5-53e67bd6b233 state_id=2023-08-01T162225--tap-shopify--target-parquet stdio=stderr 2023-08-01T162227.657700Z [info ] 'images', cmd_type=loader name=target-parquet run_id=66f5f6a0-2e97-4673-94e5-53e67bd6b233 state_id=2023-08-01T162225--tap-shopify--target-parquet stdio=stderr 2023-08-01T162227.657745Z [info ] 'options', cmd_type=loader name=target-parquet run_id=66f5f6a0-2e97-4673-94e5-53e67bd6b233 state_id=2023-08-01T162225--tap-shopify--target-parquet stdio=stderr 2023-08-01T162227.657790Z [info ] 'product_type', cmd_type=loader name=target-parquet run_id=66f5f6a0-2e97-4673-94e5-53e67bd6b233 state_id=2023-08-01T162225--tap-shopify--target-parquet stdio=stderr 2023-08-01T162227.657836Z [info ] 'published_at', cmd_type=loader name=target-parquet run_id=66f5f6a0-2e97-4673-94e5-53e67bd6b233 state_id=2023-08-01T162225--tap-shopify--target-parquet stdio=stderr 2023-08-01T162227.657883Z [info ] 'published_scope', cmd_type=loader name=target-parquet run_id=66f5f6a0-2e97-4673-94e5-53e67bd6b233 state_id=2023-08-01T162225--tap-shopify--target-parquet stdio=stderr 2023-08-01T162227.657928Z [info ] 'status', cmd_type=loader name=t…
r
So that next error looks like an error with the order schema in this case. I've made a note of this for us to get around to, but if you have some free time it might be worth you forking the repo, making the change to remove 
admin_graphql_api_id
from the 
required
blocks and see how you go. We'll gladly accept a PR back with that and any other fixes you make. Can't see why you are getting 
RECORD
messages with nothing but 
id
and 
updated
at though... We run with just 
access_token
and 
store
set with no issues.
If you do decide to fork the repo, you'll need to point your 
meltano.yml
at the local path
Copy code
pip_url: -e <path-to-tap-shopify>
and then
Copy code
meltano install --clean extractor tap-shopify
b
Thanks very much, I may not have time this week but I'd like to look into it. I am busy fixing my Clickhouse Target on the other side right now 😅
r
No problem at all, I'll let you know here if we make any more changes. 😄
Hi @ben_theunissen, we released 
v0.2.0
of 
tap-shopify
yesterday, which addresses schema issues like this (including 
admin_graphql_api_id
).
Copy code
meltano lock --update tap-shopify
meltano install --clean extractor tap-shopify
2 Views
Open in Slack
PreviousNext
Powered by