https://linen.dev logo
#plugins-general
Title
# plugins-general
j

jose_ribeiro

05/25/2021, 4:51 PM
Hi guys, I'm trying to configure the
INCREMENTAL
pull, but it seems to not be working as I'm expecting. This is my first run:
Copy code
➜  meltano-test git:(master) ✗ meltano elt tap-shopify target-bigquery --job_id=shopify_to_bq
meltano         | Running extract & load...
meltano         | Found state in extract/tap-shopify.state.json
meltano         | Found catalog in /home/zek/projects/meltano-test/extract/tap-shopify.catalog.json
tap-shopify     | INFO Syncing stream: orders
tap-shopify     | INFO GET <https://shop.myshopify.com/admin/api/2021-04/orders.json?since_id=1&updated_at_min=2021-05-25> 12:30:00+00:00&updated_at_max=2021-05-25 16:11:57+00:00&limit=175&status=any
target-bigquery | INFO Pushing state: {}
meltano         | Incremental state has been updated at 2021-05-25 16:11:58.921386.
...
target-bigquery | INFO Copy t_meltano_orders_a0856b4bf18c43d0848ec18eac403c49 to meltano_orders
target-bigquery | INFO Pushing state: {'bookmarks': {'currently_sync_stream': 'orders', 'orders': {'since_id': 3789635780666, 'updated_at': '2021-05-25T16:11:57.000000Z'}}}
meltano         | Incremental state has been updated at 2021-05-25 16:12:32.573075.
I was expecting the last state above to be the initial state on the second run bellow. But it's starting over from the
?since_id=1
when I was expecting to be
?since_id=3789635780666
. Also we can see the line
Pushing state: {}
, which is a bit strange once Meltano found the state.
Copy code
➜  meltano-test git:(master) ✗ meltano elt tap-shopify target-bigquery --job_id=shopify_to_bq
meltano         | Running extract & load...
meltano         | Found state in extract/tap-shopify.state.json
meltano         | Found catalog in /home/zek/projects/meltano-test/extract/tap-shopify.catalog.json
tap-shopify     | INFO Syncing stream: orders
tap-shopify     | INFO GET <https://shop.myshopify.com/admin/api/2021-04/orders.json?since_id=1&updated_at_min=2021-05-25> 12:30:00+00:00&updated_at_max=2021-05-25 16:14:10+00:00&limit=175&status=any
tap-shopify     | INFO --> 200 OK 2736839b
target-bigquery | INFO Pushing state: {}
meltano         | Incremental state has been updated at 2021-05-25 16:14:11.784550.
...
target-bigquery | INFO Pushing state: {'bookmarks': {'currently_sync_stream': 'orders', 'orders': {'since_id': 3789628440634, 'updated_at': '2021-05-25T16:14:10.000000Z'}}}
meltano         | Incremental state has been updated at 2021-05-25 16:14:44.962819.
meltano         | Extract & load complete!
meltano         | Transformation skipped.
On the catalog, I'm already set the replication method.
Copy code
"streams": [{
  "stream": "orders",
  "tap_stream_id": "orders",
  "schema": {
    ...
  },
  "metadata": [
    ...
  ],
  "key_properties": ["id"],
  "replication_key": "updated_at",
  "replication_method": "INCREMENTAL"
}]
Any idea of what I'm doing wrong?