jose_ribeiro
05/25/2021, 4:51 PMINCREMENTAL
pull, but it seems to not be working as I'm expecting.
This is my first run:
➜ meltano-test git:(master) ✗ meltano elt tap-shopify target-bigquery --job_id=shopify_to_bq
meltano | Running extract & load...
meltano | Found state in extract/tap-shopify.state.json
meltano | Found catalog in /home/zek/projects/meltano-test/extract/tap-shopify.catalog.json
tap-shopify | INFO Syncing stream: orders
tap-shopify | INFO GET <https://shop.myshopify.com/admin/api/2021-04/orders.json?since_id=1&updated_at_min=2021-05-25> 12:30:00+00:00&updated_at_max=2021-05-25 16:11:57+00:00&limit=175&status=any
target-bigquery | INFO Pushing state: {}
meltano | Incremental state has been updated at 2021-05-25 16:11:58.921386.
...
target-bigquery | INFO Copy t_meltano_orders_a0856b4bf18c43d0848ec18eac403c49 to meltano_orders
target-bigquery | INFO Pushing state: {'bookmarks': {'currently_sync_stream': 'orders', 'orders': {'since_id': 3789635780666, 'updated_at': '2021-05-25T16:11:57.000000Z'}}}
meltano | Incremental state has been updated at 2021-05-25 16:12:32.573075.
I was expecting the last state above to be the initial state on the second run bellow. But it's starting over from the ?since_id=1
when I was expecting to be ?since_id=3789635780666
.
Also we can see the line Pushing state: {}
, which is a bit strange once Meltano found the state.
➜ meltano-test git:(master) ✗ meltano elt tap-shopify target-bigquery --job_id=shopify_to_bq
meltano | Running extract & load...
meltano | Found state in extract/tap-shopify.state.json
meltano | Found catalog in /home/zek/projects/meltano-test/extract/tap-shopify.catalog.json
tap-shopify | INFO Syncing stream: orders
tap-shopify | INFO GET <https://shop.myshopify.com/admin/api/2021-04/orders.json?since_id=1&updated_at_min=2021-05-25> 12:30:00+00:00&updated_at_max=2021-05-25 16:14:10+00:00&limit=175&status=any
tap-shopify | INFO --> 200 OK 2736839b
target-bigquery | INFO Pushing state: {}
meltano | Incremental state has been updated at 2021-05-25 16:14:11.784550.
...
target-bigquery | INFO Pushing state: {'bookmarks': {'currently_sync_stream': 'orders', 'orders': {'since_id': 3789628440634, 'updated_at': '2021-05-25T16:14:10.000000Z'}}}
meltano | Incremental state has been updated at 2021-05-25 16:14:44.962819.
meltano | Extract & load complete!
meltano | Transformation skipped.
On the catalog, I'm already set the replication method.
"streams": [{
"stream": "orders",
"tap_stream_id": "orders",
"schema": {
...
},
"metadata": [
...
],
"key_properties": ["id"],
"replication_key": "updated_at",
"replication_method": "INCREMENTAL"
}]
Any idea of what I'm doing wrong?