Hi guys I m very new to Meltano and I m trying to pull some Meltano #plugins-general

Hi guys, I'm very new to Meltano and I'm trying to...

jose_ribeiro

05/21/2021, 7:37 PM

Hi guys, I'm very new to Meltano and I'm trying to pull some shopify data and replicate to bigquery, but it seems to be skipping all entities. Any idea why?

Copy code

$ meltano elt tap-shopify target-bigquery 
meltano         | Running extract & load...
meltano         | Found catalog in extract/tap-shopify.catalog.json
tap-shopify     | INFO Skipping stream: orders
tap-shopify     | INFO Skipping stream: collects
tap-shopify     | INFO Skipping stream: products
tap-shopify     | INFO Skipping stream: transactions
tap-shopify     | INFO Skipping stream: abandoned_checkouts
tap-shopify     | INFO Skipping stream: metafields
tap-shopify     | INFO Skipping stream: custom_collections
tap-shopify     | INFO Skipping stream: customers
tap-shopify     | INFO Skipping stream: order_refunds
tap-shopify     | INFO ----------------------
tap-shopify     | INFO ----------------------
target-bigquery | INFO Pushing state: {}
target-bigquery | INFO Pushing state: {}
meltano         | Incremental state has been updated at 2021-05-21 19:29:00.293368.
meltano         | Incremental state has been updated at 2021-05-21 19:29:00.299481.
meltano         | Extract & load complete!
meltano         | Transformation skipped.

.env

has:

Copy code

TAP_SHOPIFY_SHOP=<shop-name>
TAP_SHOPIFY_START_DATE=2021-05-01T00:00:00Z
TAP_SHOPIFY_API_KEY=<shop-key>

TARGET_BIGQUERY_PROJECT_ID=<bq-project>
TARGET_BIGQUERY_DATASET_ID=<bq-dataset>
TARGET_BIGQUERY_ADD_METADATA_COLUMNS=true
TARGET_BIGQUERY_REPLICATION_METHOD=truncate
TARGET_BIGQUERY_TABLE_PREFIX=meltano_
TARGET_BIGQUERY_PRIMARY_KEY_REQUIRED=true

TAP_SHOPIFY__CATALOG=extract/tap-shopify.catalog.json

My `meltano.yml`:

Copy code

version: 1
send_anonymous_usage_stats: true
project_id: 2d8cbfe0-514e-4dd0-8711-efbf2148c262
plugins:
  extractors:
    - name: tap-shopify
      variant: singer-io
      pip_url: tap-shopify
      select:
        - orders.*
        - transactions.*
        - products.*
  loaders:
    - name: target-bigquery
      variant: adswerve
      pip_url: git+<https://github.com/adswerve/target-bigquery.git@v0.10.2>

And I got my catalog file by running:

Copy code

meltano invoke tap-shopify --discover > extract/tap-shopify.catalog.json

Open in Slack

Previous Next