https://linen.dev logo
Join Slack
Powered by
Hello! When I'm running tap-shopify (singer) `targ...
# troubleshooting
d
Hello! When I'm running tap-shopify (singer) 
target-postgres
(transferwise) 
meltano elt tap-shopify target-postgres
Meltano creates schema and tables, but there is no data inside How to fix it? With target-jsonl I have no issues
s
@dima_petukhov I'm pretty you'll need to at least post your meltano.yml to fix your problem 😉 And btw. always try 1. "meltano invoke target-postgres" to dump to stdout 2. "meltano run tap-shopify target-postgres" as well.
d
Copy code
version: 1
default_environment: dev
project_id: 1c283f37-fbe7-4ad1-af4d-043f03d09421
environments:
  - name: dev
  - name: staging
  - name: prod
plugins:
  extractors:
    - name: tap-shopify
      variant: singer-io
      pip_url: tap-shopify
      select:
        - orders.*
        - customers.*
        - products.*
        - abandoned_checkouts.*
        - discount_codes.*
        - order_refunds.*
        - inventory_items.*
        - inventory_levels.*
        - shop.*
        - fulfillment_orders.*
        - fulfillments.*
  loaders:
    - name: target-jsonl
      variant: andyh1203
      pip_url: target-jsonl
    - name: target-postgres
      variant: transferwise
      pip_url: pipelinewise-target-postgres
  transformers:
    - name: dbt-postgres
      variant: dbt-labs
      pip_url: dbt-core~=1.3.0 dbt-postgres~=1.3.0
      config:
        host: localhost
        user: postgres
        port: 5432
        dbname: postgres
        schema: public
        add_record_metadata: true
  mappers:
    - name: meltano-map-transformer
      variant: meltano
      pip_url: git+<https://github.com/MeltanoLabs/meltano-map-transform.git>
Hello Sven here it is my config
@Sven Balnojan
a
Give the Matatika variant of tap-shopify a try? We test that with transferwise variant of target-postgres every day! Can send you a working project too if that helps.
d
@aaron_phethean Hi Aaron, yes I'm interested 🙂
@aaron_phethean Have this problem, I'm using .env config, python 3.9.16 ```2023-03-23T154622.494666Z [error ] Cannot start plugin tap-shopify: Catalog discovery failed: command ['root_path/.meltano/extractors/tap-shopify/venv/bin/tap-shopify', '--config', 'root_path/.meltano/run/tap-shopify/tap.885c5018-2704-45c6-b9d9-cc4efbfe8d05.config.json', '--state', 'root_path/.meltano/run/tap-shopify/state.json', '--discover'] returned 1 with stderr: time=2023-03-23 164622 name=tap-shopify level=INFO message=tap-shopify v0.0.1, Meltano SDK v0.4.9) time=2023-03-23 164622 name=tap-shopify level=INFO message=Skipping parse of env var settings... time=2023-03-23 164622 name=tap-shopify level=INFO message=Config validation passed with 0 warnings. time=2023-03-23 164622 name=root level=INFO message=Operator '__else__=None' was not found. Unmapped streams will be included in output. time=2023-03-23 164622 name=tap-shopify level=INFO message=Added 'inventory_items' as child stream to 'inventory_levels' time=2023-03-23 164622 name=tap-shopify level=INFO message=Added 'inventory_levels' as child stream to 'locations' time=2023-03-23 164622 name=tap-shopify level=INFO message=Added 'transactions' as child stream to 'orders' Traceback (most recent call last): File "root_path/.meltano/extractors/tap-shopify/venv/bin/tap-shopify", line 8, in <module> sys.exit(Tap_Shopify.cli()) File "root_path/.meltano/extractors/tap-shopify/venv/lib/python3.9/site-packages/click/core.py", line 1130, in call return self.main(*args, **kwargs) File "root_path/.meltano/extractors/tap-shopify/venv/lib/python3.9/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "root_path/.meltano/extractors/tap-shopify/venv/lib/python3.9/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "root_path/.meltano/extractors/tap-shopify/venv/lib/python3.9/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "root_path/.meltano/extractors/tap-shopify/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 481, in cli tap = cls( # type: ignore # Ignore 'type not callable' File "root_path/.meltano/extractors/tap-shopify/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 103, in init self.load_state(state_dict) File "root_path/.meltano/extractors/tap-shopify/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 325, in load_state for key, val in stream_state.items(): AttributeError: 'str' object has no attribute 'items' 2023-03-23T154622.495734Z [error ] Block run completed. block_type=ExtractLoadBlocks err=RunnerError('Cannot start plugin tap-shopify: Catalog discovery failed: command [\'root_path/.meltano/extractors/tap-shopify/venv/bin/tap-shopify\', \'--config\', \'root_path/.meltano/run/tap-shopify/tap.885c5018-2704-45c6-b9d9-cc4efbfe8d05.config.json\', \'--state\', \'root_path/.meltano/run/tap-shopify/state.json\', \'--discover\'] returned 1 with stderr:\n time=2023-03-23 164622 name=tap-shopify level=INFO message=tap-shopify v0.0.1, Meltano SDK v0.4.9)\ntime=2023-03-23 164622 name=tap-shopify level=INFO message=Skipping parse of env var settings...\ntime=2023-03-23 164622 name=tap-shopify level=INFO message=Config validation passed with 0 warnings.\ntime=2023-03-23 164622 name=root level=INFO message=Operator \'__else__=None\' was not found. Unmapped streams will be included in output.\ntime=2023-03-23 164622 name=tap-shopify level=INFO message=Added \'inventory_items\' as child stream to \'inventory_levels\'\ntime=2023-03-23 164622 name=tap-shopify level=INFO message=Added \'inventory_levels\' as child stream to \'locations\'\ntime=2023-03-23 164622 name=tap-shopify level=INFO message=Added \'transactions\' as child stream to \'orders\'\nTraceback (most recent call last):\n File "root_path/.meltano/extractors/tap-shopify/venv/bin/tap-shop…
a
Thoughts on this @daniel_walker ? On my phone and a bit hard to read the logs!
e
Might be useful to inspect the contents of 
root_path/.meltano/run/tap-shopify/state.json
d
@edgar_ramirez_mondragon
Copy code
{
  "bookmarks": {
    "currently_sync_stream": "orders",
    "orders": {
      "updated_at": "2019-09-05T00:00:00.000000Z"
    }
  }
}
If I relaunch command (I've changed shopify to Matatika) 
meltano elt tap-shopify target-postgres
the output: 1/3 ```2023-03-23T160400.285294Z [info ] Running extract & load... name=meltano run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres 2023-03-23T160401.227346Z [warning ] No state was found, complete import. 2023-03-23T160402.499647Z [info ] time=2023-03-23 170402 name=tap-shopify level=INFO message=tap-shopify v0.0.1, Meltano SDK v0.4.9) cmd_type=extractor name=tap-shopify run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160402.500790Z [info ] time=2023-03-23 170402 name=tap-shopify level=INFO message=Skipping parse of env var settings... cmd_type=extractor name=tap-shopify run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160402.501390Z [info ] time=2023-03-23 170402 name=tap-shopify level=INFO message=Config validation passed with 0 warnings. cmd_type=extractor name=tap-shopify run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160402.507043Z [info ] time=2023-03-23 170402 name=root level=INFO message=Operator '__else__=None' was not found. Unmapped streams will be included in output. cmd_type=extractor name=tap-shopify run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160402.516905Z [info ] time=2023-03-23 170402 name=tap-shopify level=INFO message=Added 'inventory_items' as child stream to 'inventory_levels' cmd_type=extractor name=tap-shopify run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160402.517178Z [info ] time=2023-03-23 170402 name=tap-shopify level=INFO message=Added 'inventory_levels' as child stream to 'locations' cmd_type=extractor name=tap-shopify run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160402.517434Z [info ] time=2023-03-23 170402 name=tap-shopify level=INFO message=Added 'transactions' as child stream to 'orders' cmd_type=extractor name=tap-shopify run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160402.517723Z [info ] time=2023-03-23 170402 name=tap-shopify level=INFO message=Beginning incremental sync of 'abandoned_checkouts'... cmd_type=extractor name=tap-shopify run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160402.517929Z [info ] time=2023-03-23 170402 name=tap-shopify level=INFO message=Tap has custom mapper. Using 1 provided map(s). cmd_type=extractor name=tap-shopify run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160402.628043Z [info ] time=2023-03-23 170402 name=target_postgres level=INFO message=Schema 'tap_shopify' does not exist. Creating... CREATE SCHEMA IF NOT EXISTS tap_shopify cmd_type=loader name=target-postgres run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160402.666936Z [info ] time=2023-03-23 170402 name=target_postgres level=INFO message=Table '"abandoned_checkouts"' does not exist. Creating... CREATE TABLE IF NOT EXISTS tap_shopify."abandoned_checkouts" ("abandoned_checkout_url" character varying, "billing_address" jsonb, "buyer_accepts_marketing" boolean, "buyer_accepts_sms_marketing" boolean, "cart_token" character varying, "closed_at" character varying, "completed_at" character varying, "created_at" character varying, "currency" jsonb, "customer" jsonb, "customer_l…
2/3 ```2023-03-23T160403.998972Z [info ] time=2023-03-23 170403 name=tap-shopify level=WARNING message=Property 'source_identifier' was present in the 'abandoned_checkouts' stream but not found in catalog schema. Ignoring. cmd_type=extractor name=tap-shopify run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160403.999199Z [info ] time=2023-03-23 170403 name=tap-shopify level=WARNING message=Property 'source_url' was present in the 'abandoned_checkouts' stream but not found in catalog schema. Ignoring. cmd_type=extractor name=tap-shopify run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160403.999378Z [info ] time=2023-03-23 170403 name=tap-shopify level=WARNING message=Property 'source' was present in the 'abandoned_checkouts' stream but not found in catalog schema. Ignoring. cmd_type=extractor name=tap-shopify run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160403.999789Z [info ] time=2023-03-23 170403 name=tap-shopify level=WARNING message=Property 'name' was present in the 'abandoned_checkouts' stream but not found in catalog schema. Ignoring. cmd_type=extractor name=tap-shopify run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160405.164280Z [info ] time=2023-03-23 170405 name=tap-shopify level=INFO message=INFO METRIC: {'type': 'timer', 'metric': 'http_request_duration', 'value': 1.118467, 'tags': {'endpoint': '/api/2022-01/checkouts.json', 'http_status_code': 200, 'status': 'succeeded'}} cmd_type=extractor name=tap-shopify run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160406.063439Z [info ] time=2023-03-23 170406 name=tap-shopify level=INFO message=INFO METRIC: {'type': 'timer', 'metric': 'http_request_duration', 'value': 0.858548, 'tags': {'endpoint': '/api/2022-01/checkouts.json', 'http_status_code': 200, 'status': 'succeeded'}} cmd_type=extractor name=tap-shopify run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160406.094192Z [info ] Traceback (most recent call last): cmd_type=extractor name=tap-shopify run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160406.094578Z [info ] File "root_path/.meltano/extractors/tap-shopify/venv/bin/tap-shopify", line 8, in <module> cmd_type=extractor name=tap-shopify run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160406.095137Z [info ] sys.exit(Tap_Shopify.cli()) cmd_type=extractor name=tap-shopify run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160406.095344Z [info ] File "root_path/.meltano/extractors/tap-shopify/venv/lib/python3.9/site-packages/click/core.py", line 1130, in call cmd_type=extractor name=tap-shopify run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160406.095700Z [info ] return self.main(*args, **kwargs) cmd_type=extractor name=tap-shopify run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160406.095888Z [info ] File "root_path/.meltano/extractors/tap-shopify/venv/lib/python3.9/site-packages/click/core.py", line 1055, in main cmd_type=extractor name=tap-shopify run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160406.096158Z [info ] rv = self.invoke(ctx) …
3/3 ```2023-03-23T160406.262362Z [info ] File "root_path/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/target_postgres/__init__.py", line 373, in main cmd_type=loader name=target-postgres run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160406.262782Z [info ] persist_lines(config, singer_messages) cmd_type=loader name=target-postgres run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160406.263221Z [info ] File "root_path/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/target_postgres/__init__.py", line 239, in persist_lines cmd_type=loader name=target-postgres run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160406.263756Z [info ] flushed_state = flush_streams(records_to_load, row_count, stream_to_sync, config, state, flushed_state) cmd_type=loader name=target-postgres run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160406.264001Z [info ] File "root_path/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/target_postgres/__init__.py", line 288, in flush_streams cmd_type=loader name=target-postgres run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160406.264206Z [info ] Parallel()(delayed(load_stream_batch)( cmd_type=loader name=target-postgres run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160406.264463Z [info ] File "root_path/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/joblib/parallel.py", line 1085, in call cmd_type=loader name=target-postgres run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160406.264914Z [info ] if self.dispatch_one_batch(iterator): cmd_type=loader name=target-postgres run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160406.265141Z [info ] File "root_path/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/joblib/parallel.py", line 901, in dispatch_one_batch cmd_type=loader name=target-postgres run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160406.265588Z [info ] self._dispatch(tasks) cmd_type=loader name=target-postgres run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160406.265920Z [info ] File "root_path/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/joblib/parallel.py", line 819, in _dispatch cmd_type=loader name=target-postgres run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160406.266113Z [info ] job = self._backend.apply_async(batch, callback=cb) cmd_type=loader name=target-postgres run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160406.266331Z [info ] File "root_path/.meltano/loaders/target-postgres/venv/lib/python3.9/site-packages/joblib/_parallel_backends.py", line 208, in apply_async cmd_type=loader name=target-postgres run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T160406.266756Z [info ] result = ImmediateResult(func) cmd_type=loader name=target-postgres run_id=740d32da-25e3-4c20-b1ee-13f2298c97f5 state_id=2023-03-23T160358--tap-shopify--target-postgres stdio=stderr 2023-03-23T16:…
e
Loop detected in pagination.
Seems like a bug in the tap, I think.
r
Yeah, we'll have a look into that ASAP. @aaron_phethean
d
updated meltano.yml
Copy code
version: 1
default_environment: dev
project_id: 1c283f37-fbe7-4ad1-af4d-043f03d09421
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-shopify
    variant: matatika
    pip_url: git+<https://github.com/Matatika/tap-shopify.git>
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl
  - name: target-postgres
    variant: transferwise
    pip_url: pipelinewise-target-postgres
  transformers:
  - name: dbt-postgres
    variant: dbt-labs
    pip_url: dbt-core~=1.3.0 dbt-postgres~=1.3.0
    config:
      host: localhost
      user: postgres
      port: 5432
      dbname: postgres
      schema: public
      add_record_metadata: true
  mappers:
  - name: meltano-map-transformer
    variant: meltano
    pip_url: git+<https://github.com/MeltanoLabs/meltano-map-transform.git>
a
This might be a silly question, but seems worth checking. Did you ‘meltano install’ after changing the tap? Be a shame to investigate an issue only to find we’re running the code from another tap.
d
I've done only this 
meltano remove extractor tap-shopify
then 
meltano add extractor tap-shopify --variant=matatika
a
One more thing. Any reason you are using ‘meltano elt…’ instead of ‘meltano run…’?
d
There is no reason
Is there difference between?
r
meltano remove
will remove the associated virtual environment, and 
meltano add
will implicitly install the plugin by default - so pretty sure it is our variant running in the case of the second set of logs.
a
‘elt’ is deprecated. ‘run’ will automatically create a state key and run with state. Otherwise should be the same.
d
I got it, even with 'run' it gives me an error ```2023-03-23T163914.504248Z [info ] Environment 'dev' is active 2023-03-23T163918.228160Z [error ] Cannot start plugin tap-shopify: Catalog discovery failed: command ['root_path/.meltano/extractors/tap-shopify/venv/bin/tap-shopify', '--config', 'root_path/.meltano/run/tap-shopify/tap.193528ab-1919-45e5-aa4d-35d0022066fe.config.json', '--state', 'root_path/.meltano/run/tap-shopify/state.json', '--discover'] returned 1 with stderr: time=2023-03-23 173918 name=tap-shopify level=INFO message=tap-shopify v0.0.1, Meltano SDK v0.4.9) time=2023-03-23 173918 name=tap-shopify level=INFO message=Skipping parse of env var settings... time=2023-03-23 173918 name=tap-shopify level=INFO message=Config validation passed with 0 warnings. time=2023-03-23 173918 name=root level=INFO message=Operator '__else__=None' was not found. Unmapped streams will be included in output. time=2023-03-23 173918 name=tap-shopify level=INFO message=Added 'inventory_items' as child stream to 'inventory_levels' time=2023-03-23 173918 name=tap-shopify level=INFO message=Added 'inventory_levels' as child stream to 'locations' time=2023-03-23 173918 name=tap-shopify level=INFO message=Added 'transactions' as child stream to 'orders' Traceback (most recent call last): File "root_path/.meltano/extractors/tap-shopify/venv/bin/tap-shopify", line 8, in <module> sys.exit(Tap_Shopify.cli()) File "root_path/.meltano/extractors/tap-shopify/venv/lib/python3.9/site-packages/click/core.py", line 1130, in call return self.main(*args, **kwargs) File "root_path/.meltano/extractors/tap-shopify/venv/lib/python3.9/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "root_path/.meltano/extractors/tap-shopify/venv/lib/python3.9/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "root_path/.meltano/extractors/tap-shopify/venv/lib/python3.9/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "root_path/.meltano/extractors/tap-shopify/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 481, in cli tap = cls( # type: ignore # Ignore 'type not callable' File "root_path/.meltano/extractors/tap-shopify/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 103, in init self.load_state(state_dict) File "root_path/.meltano/extractors/tap-shopify/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 325, in load_state for key, val in stream_state.items(): AttributeError: 'str' object has no attribute 'items' 2023-03-23T163918.229292Z [error ] Block run completed. block_type=ExtractLoadBlocks err=RunnerError('Cannot start plugin tap-shopify: Catalog discovery failed: command [\'root_path/.meltano/extractors/tap-shopify/venv/bin/tap-shopify\', \'--config\', \'root_path/.meltano/run/tap-shopify/tap.193528ab-1919-45e5-aa4d-35d0022066fe.config.json\', \'--state\', \'root_path/.meltano/run/tap-shopify/state.json\', \'--discover\'] returned 1 with stderr:\n time=2023-03-23 173918 name=tap-shopify level=INFO message=tap-shopify v0.0.1, Meltano SDK v0.4.9)\ntime=2023-03-23 173918 name=tap-shopify level=INFO message=Skipping parse of env var settings...\ntime=2023-03-23 173918 name=tap-shopify level=INFO message=Config validation passed with 0 warnings.\ntime=2023-03-23 173918 name=root level=INFO message=Operator \'__else__=None\' was not found. Unmapped streams will be included in output.\ntime=2023-03-23 173918 name=tap-shopify level=INFO message=Added \'inventory_items\' as child stream to \'inventory_levels\'\ntime=2023-03-23 173918 name=tap-shopify level=INFO message=Added \'inventory_levels\' as child stream to \'locations\'\ntime=2023-03-23 173918 name=tap-shopify level=INFO message=Added \'transactions\' as child stream to \'orders\'\nTraceback (most recent call last):\n File "root_path/.melt…
@aaron_phethean I could provide you more details if you need
a
Great. Let’s see what the guys find and we’ll ping you. Reproducing it is always the biggest step! At this moment we’re looking at a tap-shopify issue, but you actually started with an issue in the target! Or some data that caused an issue. Going back to @Sven Balnojan’s recommendation, when debugging it’s possible to isolate the problem: 1) ‘meltano invoke tap-shopify > data.out’ 2) ‘cat data.out | meltano invoke target-postgres’ Step 1 means the tap works. Step 2 means the target works.
d
```»»»» meltano invoke tap-shopify > data.out 0|174858 2023-03-23T165244.579802Z [info ] Environment 'dev' is active time=2023-03-23 175248 name=tap-shopify level=INFO message=tap-shopify v0.0.1, Meltano SDK v0.4.9) time=2023-03-23 175248 name=tap-shopify level=INFO message=Skipping parse of env var settings... time=2023-03-23 175248 name=tap-shopify level=INFO message=Config validation passed with 0 warnings. time=2023-03-23 175248 name=root level=INFO message=Operator '__else__=None' was not found. Unmapped streams will be included in output. time=2023-03-23 175248 name=tap-shopify level=INFO message=Added 'inventory_items' as child stream to 'inventory_levels' time=2023-03-23 175248 name=tap-shopify level=INFO message=Added 'inventory_levels' as child stream to 'locations' time=2023-03-23 175248 name=tap-shopify level=INFO message=Added 'transactions' as child stream to 'orders' time=2023-03-23 175248 name=tap-shopify level=INFO message=Beginning incremental sync of 'abandoned_checkouts'... time=2023-03-23 175248 name=tap-shopify level=INFO message=Tap has custom mapper. Using 1 provided map(s). time=2023-03-23 175249 name=tap-shopify level=INFO message=INFO METRIC: {'type': 'timer', 'metric': 'http_request_duration', 'value': 1.086029, 'tags': {'endpoint': '/api/2022-01/checkouts.json', 'http_status_code': 200, 'status': 'succeeded'}} time=2023-03-23 175249 name=tap-shopify level=WARNING message=Property 'note_attributes' was present in the 'abandoned_checkouts' stream but not found in catalog schema. Ignoring. time=2023-03-23 175249 name=tap-shopify level=WARNING message=Property 'source_identifier' was present in the 'abandoned_checkouts' stream but not found in catalog schema. Ignoring. time=2023-03-23 175249 name=tap-shopify level=WARNING message=Property 'source_url' was present in the 'abandoned_checkouts' stream but not found in catalog schema. Ignoring. time=2023-03-23 175249 name=tap-shopify level=WARNING message=Property 'source' was present in the 'abandoned_checkouts' stream but not found in catalog schema. Ignoring. time=2023-03-23 175249 name=tap-shopify level=WARNING message=Property 'name' was present in the 'abandoned_checkouts' stream but not found in catalog schema. Ignoring. time=2023-03-23 175250 name=tap-shopify level=INFO message=INFO METRIC: {'type': 'timer', 'metric': 'http_request_duration', 'value': 0.875022, 'tags': {'endpoint': '/api/2022-01/checkouts.json', 'http_status_code': 200, 'status': 'succeeded'}} time=2023-03-23 175251 name=tap-shopify level=INFO message=INFO METRIC: {'type': 'timer', 'metric': 'http_request_duration', 'value': 0.990351, 'tags': {'endpoint': '/api/2022-01/checkouts.json', 'http_status_code': 200, 'status': 'succeeded'}} Traceback (most recent call last): File "root_path/.meltano/extractors/tap-shopify/venv/bin/tap-shopify", line 8, in <module> sys.exit(Tap_Shopify.cli()) File "root_path/.meltano/extractors/tap-shopify/venv/lib/python3.9/site-packages/click/core.py", line 1130, in call return self.main(*args, **kwargs) File "root_path/.meltano/extractors/tap-shopify/venv/lib/python3.9/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "root_path/.meltano/extractors/tap-shopify/venv/lib/python3.9/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "root_path/.meltano/extractors/tap-shopify/venv/lib/python3.9/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "root_path/.meltano/extractors/tap-shopify/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 499, in cli tap.sync_all() File "root_path/.meltano/extractors/tap-shopify/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 379, in sync_all …
e
that’s progress! you’re missing config for either 
default_target_schema
or 
schema_mapping
in target-postgres
r
Hi @dima_petukhov, I just merged in a fix for that error you were seeing - here is the PR if you were interested. I have tested the change against our Shopify test store, but since we don't use the platform in any real capacity there is only so much data we can mock - as in this case, clearly some scenarios are not full tested. Give it a try and let us know how it goes! 😄 Since our 
tap-shopify
in not yet pinned to a specific version on Meltano Hub, you should just have to
Copy code
meltano install extractor tap-shopify --clean
and then (as before)
Copy code
meltano invoke tap-shopify > data.out
d
@edgar_ramirez_mondragon if we are not setting default_target_schema, target-posgtres will create it with 
tap-shopify
name as I understand?
@Reuben (Matatika) there is an error 1/2 ```»»»» meltano invoke tap-shopify > data.out; cat data.out | meltano invoke target-postgres; 130|093738 2023-03-24T083757.151420Z [info ] Environment 'dev' is active time=2023-03-24 093800 name=tap-shopify level=INFO message=tap-shopify v0.0.1, Meltano SDK v0.4.9) time=2023-03-24 093800 name=tap-shopify level=INFO message=Skipping parse of env var settings... time=2023-03-24 093800 name=tap-shopify level=INFO message=Config validation passed with 0 warnings. time=2023-03-24 093800 name=root level=INFO message=Operator '__else__=None' was not found. Unmapped streams will be included in output. time=2023-03-24 093800 name=tap-shopify level=INFO message=Added 'inventory_items' as child stream to 'inventory_levels' time=2023-03-24 093800 name=tap-shopify level=INFO message=Added 'inventory_levels' as child stream to 'locations' time=2023-03-24 093800 name=tap-shopify level=INFO message=Added 'transactions' as child stream to 'orders' time=2023-03-24 093800 name=tap-shopify level=INFO message=Beginning incremental sync of 'abandoned_checkouts'... time=2023-03-24 093800 name=tap-shopify level=INFO message=Tap has custom mapper. Using 1 provided map(s). time=2023-03-24 093801 name=tap-shopify level=INFO message=INFO METRIC: {'type': 'timer', 'metric': 'http_request_duration', 'value': 1.052793, 'tags': {'endpoint': '/api/2022-01/checkouts.json', 'http_status_code': 200, 'status': 'succeeded'}} time=2023-03-24 093801 name=tap-shopify level=WARNING message=Property 'note_attributes' was present in the 'abandoned_checkouts' stream but not found in catalog schema. Ignoring. time=2023-03-24 093801 name=tap-shopify level=WARNING message=Property 'source_identifier' was present in the 'abandoned_checkouts' stream but not found in catalog schema. Ignoring. time=2023-03-24 093801 name=tap-shopify level=WARNING message=Property 'source_url' was present in the 'abandoned_checkouts' stream but not found in catalog schema. Ignoring. time=2023-03-24 093801 name=tap-shopify level=WARNING message=Property 'source' was present in the 'abandoned_checkouts' stream but not found in catalog schema. Ignoring. time=2023-03-24 093801 name=tap-shopify level=WARNING message=Property 'name' was present in the 'abandoned_checkouts' stream but not found in catalog schema. Ignoring. time=2023-03-24 093802 name=tap-shopify level=INFO message=INFO METRIC: {'type': 'timer', 'metric': 'http_request_duration', 'value': 0.849928, 'tags': {'endpoint': '/api/2022-01/checkouts.json', 'http_status_code': 200, 'status': 'succeeded'}} time=2023-03-24 093803 name=tap-shopify level=INFO message=INFO METRIC: {'type': 'timer', 'metric': 'http_request_duration', 'value': 0.793887, 'tags': {'endpoint': '/api/2022-01/checkouts.json', 'http_status_code': 200, 'status': 'succeeded'}} time=2023-03-24 093803 name=tap-shopify level=INFO message=INFO METRIC: {'type': 'counter', 'metric': 'record_count', 'value': 143, 'tags': {'stream': 'abandoned_checkouts'}} time=2023-03-24 093803 name=tap-shopify level=INFO message=Beginning incremental sync of 'collects'... time=2023-03-24 093803 name=tap-shopify level=INFO message=Tap has custom mapper. Using 1 provided map(s). time=2023-03-24 093803 name=tap-shopify level=INFO message=INFO METRIC: {'type': 'timer', 'metric': 'http_request_duration', 'value': 0.111708, 'tags': {'endpoint': '/api/2022-01/collects.json', 'http_status_code': 200, 'status': 'succeeded'}} time=2023-03-24 093803 name=tap-shopify level=INFO message=INFO METRIC: {'type': 'timer', 'metric': 'http_request_duration', 'value': 0.130306, 'tags': {'endpoint': '/api/2022-01/collects.json', 'http_status_code': 200, 'status': 'succeeded'}} time=2023-03-24 093803 name=tap-shopify level=INFO message=INFO METRIC: {'type': 'counter', 'metric': 'record_count', 'value': 58, 'tags': {'stream': 'collects'}} time=2023-03-24 093803 name=tap-shopify level=INFO message…
2/2 ```{'endpoint': '/api/2022-01/customers.json', 'http_status_code': 200, 'status': 'succeeded'}} time=2023-03-24 093815 name=tap-shopify level=INFO message=INFO METRIC: {'type': 'timer', 'metric': 'http_request_duration', 'value': 0.613278, 'tags': {'endpoint': '/api/2022-01/customers.json', 'http_status_code': 200, 'status': 'succeeded'}} time=2023-03-24 093816 name=tap-shopify level=INFO message=INFO METRIC: {'type': 'timer', 'metric': 'http_request_duration', 'value': 0.558327, 'tags': {'endpoint': '/api/2022-01/customers.json', 'http_status_code': 200, 'status': 'succeeded'}} time=2023-03-24 093817 name=tap-shopify level=INFO message=INFO METRIC: {'type': 'timer', 'metric': 'http_request_duration', 'value': 0.529522, 'tags': {'endpoint': '/api/2022-01/customers.json', 'http_status_code': 200, 'status': 'succeeded'}} time=2023-03-24 093818 name=tap-shopify level=INFO message=INFO METRIC: {'type': 'timer', 'metric': 'http_request_duration', 'value': 0.609155, 'tags': {'endpoint': '/api/2022-01/customers.json', 'http_status_code': 200, 'status': 'succeeded'}} time=2023-03-24 093818 name=tap-shopify level=INFO message=INFO METRIC: {'type': 'timer', 'metric': 'http_request_duration', 'value': 0.530791, 'tags': {'endpoint': '/api/2022-01/customers.json', 'http_status_code': 200, 'status': 'succeeded'}} time=2023-03-24 093819 name=tap-shopify level=INFO message=INFO METRIC: {'type': 'timer', 'metric': 'http_request_duration', 'value': 0.640411, 'tags': {'endpoint': '/api/2022-01/customers.json', 'http_status_code': 200, 'status': 'succeeded'}} time=2023-03-24 093820 name=tap-shopify level=INFO message=INFO METRIC: {'type': 'timer', 'metric': 'http_request_duration', 'value': 1.081194, 'tags': {'endpoint': '/api/2022-01/customers.json', 'http_status_code': 200, 'status': 'succeeded'}} time=2023-03-24 093821 name=tap-shopify level=INFO message=INFO METRIC: {'type': 'timer', 'metric': 'http_request_duration', 'value': 1.031811, 'tags': {'endpoint': '/api/2022-01/customers.json', 'http_status_code': 200, 'status': 'succeeded'}} time=2023-03-24 093821 name=tap-shopify level=INFO message=INFO METRIC: {'type': 'timer', 'metric': 'http_request_duration', 'value': 0.20683, 'tags': {'endpoint': '/api/2022-01/customers.json', 'http_status_code': 200, 'status': 'succeeded'}} time=2023-03-24 093821 name=tap-shopify level=INFO message=INFO METRIC: {'type': 'counter', 'metric': 'record_count', 'value': 1354, 'tags': {'stream': 'customers'}} time=2023-03-24 093821 name=tap-shopify level=INFO message=Beginning full_table sync of 'locations'... time=2023-03-24 093821 name=tap-shopify level=INFO message=Tap has custom mapper. Using 1 provided map(s). time=2023-03-24 093821 name=tap-shopify level=INFO message=INFO METRIC: {'type': 'timer', 'metric': 'http_request_duration', 'value': 0.118128, 'tags': {'endpoint': '/api/2022-01/locations.json', 'http_status_code': 200, 'status': 'succeeded'}} time=2023-03-24 093821 name=tap-shopify level=INFO message=Beginning full_table sync of 'inventory_levels' with context: {'location_id': 71637139669}... time=2023-03-24 093821 name=tap-shopify level=INFO message=Tap has custom mapper. Using 1 provided map(s). time=2023-03-24 093821 name=tap-shopify level=INFO message=INFO METRIC: {'type': 'timer', 'metric': 'http_request_duration', 'value': 0.107497, 'tags': {'endpoint': '/api/2022-01/inventory_levels.json?location_ids={location_id}', 'http_status_code': 200, 'status': 'succeeded', 'context': {'location_id': 71637139669}}} time=2023-03-24 093821 name=tap-shopify level=INFO message=INFO METRIC: {'type': 'counter', 'metric': 'record_count', 'value': 0, 'tags': {'stream': 'inventory_levels', 'context': {'location_id': 71637139669}}} time=2023-03-24 093821 name=tap-shopify level=WARNING message=Property 'country_name' was present in the 'locations' stream but not found in catalog schema. Ignoring. time=2023-03-24 093821 name=tap-shopify level=WARNING message=Property 'admin_g…
Guys I will switch to another solution, thank you
Open in Slack
PreviousNext
Powered by