Hi guys I m having a funny behavior happening on my ELT I m Meltano #troubleshooting

Hi guys, I'm having a funny behavior happening on ...

jose_ribeiro

07/13/2021, 2:02 PM

Hi guys, I'm having a funny behavior happening on my ELT. I'm using the

tap-shopify

target-bigquery

orchestrator-airflow

. I have multiple machines and I'm trying to pull

products

transactions

orders

and

refunds

. At machine one, I'm only getting

products

orders

At the second machine, I'm getting only the

orders

. I've checked my catalog, and I'm correctly "selecting" the entities that I wanna pull. I checked the logs at

.meltano/logs/elt/shopify-to-bigquery/*/*.log

and the entities are not being pulled. Does someone had a similar behavior? Any clue about what could be happening?

jose_ribeiro

07/13/2021, 2:03 PM

that's my

.env

Copy code

# .env
TAP_SHOPIFY_SHOP=<shopname>
TAP_SHOPIFY_API_KEY=<password>

TARGET_BIGQUERY_PROJECT_ID=<glcoud-project>
TARGET_BIGQUERY_DATASET_ID=<glcoud-dataseet>
TARGET_BIGQUERY_TABLE_PREFIX=_meltano_

TAP_SHOPIFY_START_DATE=2017-01-01T00:00:00Z
TARGET_BIGQUERY_ADD_METADATA_COLUMNS=true
TARGET_BIGQUERY_REPLICATION_METHOD=append
TAP_SHOPIFY__CATALOG=extract/tap-shopify.catalog.json

meltano.yml

Copy code

# meltano.yml
version: 1
send_anonymous_usage_stats: true
project_id: hash-project-id
plugins:
  extractors:
  - name: tap-shopify
    variant: singer-io
    pip_url: tap-shopify
  loaders:
  - name: target-bigquery
    variant: adswerve
    pip_url: git+<https://github.com/adswerve/target-bigquery.git@v0.10.2>
  orchestrators:
  - name: airflow
    pip_url: apache-airflow==1.10.14 --constraint <https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.6.txt>
  files:
  - name: airflow
    pip_url: git+<https://gitlab.com/meltano/files-airflow.git>
schedules:
- name: shopify-to-bigquery
  extractor: tap-shopify
  loader: target-bigquery
  transform: skip
  interval: '@hourly'
  start_date: 2017-01-01 00:00:00

Open in Slack

Previous Next