https://linen.dev logo
#plugins-general
Title
# plugins-general
j

jose_ribeiro

05/21/2021, 7:37 PM
Hi guys, I'm very new to Meltano and I'm trying to pull some shopify data and replicate to bigquery, but it seems to be skipping all entities. Any idea why?
Copy code
$ meltano elt tap-shopify target-bigquery 
meltano         | Running extract & load...
meltano         | Found catalog in extract/tap-shopify.catalog.json
tap-shopify     | INFO Skipping stream: orders
tap-shopify     | INFO Skipping stream: collects
tap-shopify     | INFO Skipping stream: products
tap-shopify     | INFO Skipping stream: transactions
tap-shopify     | INFO Skipping stream: abandoned_checkouts
tap-shopify     | INFO Skipping stream: metafields
tap-shopify     | INFO Skipping stream: custom_collections
tap-shopify     | INFO Skipping stream: customers
tap-shopify     | INFO Skipping stream: order_refunds
tap-shopify     | INFO ----------------------
tap-shopify     | INFO ----------------------
target-bigquery | INFO Pushing state: {}
target-bigquery | INFO Pushing state: {}
meltano         | Incremental state has been updated at 2021-05-21 19:29:00.293368.
meltano         | Incremental state has been updated at 2021-05-21 19:29:00.299481.
meltano         | Extract & load complete!
meltano         | Transformation skipped.
My
.env
has:
Copy code
TAP_SHOPIFY_SHOP=<shop-name>
TAP_SHOPIFY_START_DATE=2021-05-01T00:00:00Z
TAP_SHOPIFY_API_KEY=<shop-key>

TARGET_BIGQUERY_PROJECT_ID=<bq-project>
TARGET_BIGQUERY_DATASET_ID=<bq-dataset>
TARGET_BIGQUERY_ADD_METADATA_COLUMNS=true
TARGET_BIGQUERY_REPLICATION_METHOD=truncate
TARGET_BIGQUERY_TABLE_PREFIX=meltano_
TARGET_BIGQUERY_PRIMARY_KEY_REQUIRED=true

TAP_SHOPIFY__CATALOG=extract/tap-shopify.catalog.json
My `meltano.yml`:
Copy code
version: 1
send_anonymous_usage_stats: true
project_id: 2d8cbfe0-514e-4dd0-8711-efbf2148c262
plugins:
  extractors:
    - name: tap-shopify
      variant: singer-io
      pip_url: tap-shopify
      select:
        - orders.*
        - transactions.*
        - products.*
  loaders:
    - name: target-bigquery
      variant: adswerve
      pip_url: git+<https://github.com/adswerve/target-bigquery.git@v0.10.2>
And I got my catalog file by running:
Copy code
meltano invoke tap-shopify --discover > extract/tap-shopify.catalog.json