alex_dimov
11/25/2022, 11:21 AMThe context object SHOULD NOT be mutated during the stream’s lifecycle.Then how do I store the id of the last received entry?
Reuben (Matatika)
11/25/2022, 12:30 PMlog_id
from state and using it to set some URL params. All I needed to do was set the replication_key
and Meltano handles all state management.
https://sdk.meltano.com/en/latest/implementation/state.htmlalex_dimov
11/25/2022, 12:32 PMReuben (Matatika)
11/25/2022, 12:32 PMnext_page_token
so it checks state for a log_id
to start from.alex_dimov
11/25/2022, 12:34 PMReuben (Matatika)
11/25/2022, 12:37 PMalex_dimov
11/25/2022, 12:38 PMalex_dimov
11/25/2022, 5:35 PMalex_dimov
11/25/2022, 6:02 PM[
{
log_id: 1
dateTime: "11-10-2022T12:03:00"
},
{
log_id: 2
dateTime: "11-10-2022T12:04:00"
},
{
log_id: 3
dateTime: "11-10-2022T12:06:00"
},
]
Then this function will return 1. (although, in theory, get_next_page_token should have returned 3 in order to get all records where the id is greater than the last received in the next request)Reuben (Matatika)
11/25/2022, 6:29 PMIn your example (oauth0), you got the value from the context_state.get("replication_key_value") field, but I do the same and for some reason I have another nested progress_markers object. What am I doing wrong?Not sure. This is my state for
meltano run tap-auth0 target-jsonl
with only stream_auth0_logs.*
selected:Reuben (Matatika)
11/25/2022, 6:39 PMThen this function will return 1. (although, in theory, get_next_page_token should have returned 3 in order to get all records where the id is greater than the last received in the next request)This is just an Auth0-specific implementation. If there is no
next_page_token
or log_id
state to start from, get the single oldest available log and begin pagination from there.alex_dimov
11/28/2022, 12:48 PM