jose_ribeiro
05/24/2021, 10:28 PMmeltano elt tap-shopify target-bigquery --full-refresh --job_id=shopify_to_bq
On my .env
, I'm specifying the state and the catalog like:
TAP_SHOPIFY__CATALOG=extract/tap-shopify.catalog.json
TAP_SHOPIFY__STATE=extract/tap-shopify.state.json
On my catalog, I have the replication method, and key as:
"metadata": {
"selected": true,
"table-key-properties": ["id"],
"forced-replication-method": "INCREMENTAL",
"valid-replication-keys": ["updated_at"]
}
I can see the bookmark line on console, but the state file remains empty.
{
"bookmarks": {
"currently_sync_stream": "transactions",
"orders": {
"since_id": 123,
"updated_at": "2021-05-24T22:18:48.000000Z"
},
"products": {
"since_id": 456,
"updated_at": "2021-05-24T22:18:55.000000Z"
},
"transaction_orders": {
"since_id": 789,
"updated_at": "2021-05-24T22:19:20.000000Z"
},
"transactions": {
"created_at": "2021-05-24T22:17:37.000000Z"
}
}
}
Also, if I write the state file using the content above, it also ignores that and start over again using the since_id=0
.
Any idea how to fix or what I'm doing wrong?