https://linen.dev logo
#plugins-general
Title
# plugins-general
j

jose_ribeiro

05/24/2021, 10:28 PM
Hi, I'm running the tap-shopify and I'm facing a weird behavior, which I couldn't find a solution for yet. The elt run seems to not be respecting the bookmark, so, every time that I run the elt, I got duplicated records on BQ. I'm running the elt using:
Copy code
meltano elt tap-shopify target-bigquery --full-refresh --job_id=shopify_to_bq
On my
.env
, I'm specifying the state and the catalog like:
Copy code
TAP_SHOPIFY__CATALOG=extract/tap-shopify.catalog.json
TAP_SHOPIFY__STATE=extract/tap-shopify.state.json
On my catalog, I have the replication method, and key as:
Copy code
"metadata": {
  "selected": true,
  "table-key-properties": ["id"],
  "forced-replication-method": "INCREMENTAL",
  "valid-replication-keys": ["updated_at"]
}
I can see the bookmark line on console, but the state file remains empty.
Copy code
{
  "bookmarks": {
    "currently_sync_stream": "transactions",
    "orders": {
      "since_id": 123,
      "updated_at": "2021-05-24T22:18:48.000000Z"
    },
    "products": {
      "since_id": 456,
      "updated_at": "2021-05-24T22:18:55.000000Z"
    },
    "transaction_orders": {
      "since_id": 789,
      "updated_at": "2021-05-24T22:19:20.000000Z"
    },
    "transactions": {
      "created_at": "2021-05-24T22:17:37.000000Z"
    }
  }
}
Also, if I write the state file using the content above, it also ignores that and start over again using the
since_id=0
. Any idea how to fix or what I'm doing wrong?