<@U06D0USC2CQ> Good day. In your example i have se...
# singer-tap-development
a
@Reuben (Matatika) Good day. In your example i have seen that you use context_state. Can you share me some dock or explain context_state (how it work?) How often is this storage reset, what is the likelihood that I will start storing duplicate events? I have found this Context Object, but i don't understand why i can't change it
The context object SHOULD NOT be mutated during the stream’s lifecycle.
Then how do I store the id of the last received entry?
r
This was a while ago so I'm not sure if this the "correct" way to do things, but all that is doing is getting the last encountered
log_id
from state and using it to set some URL params. All I needed to do was set the
replication_key
and Meltano handles all state management. https://sdk.meltano.com/en/latest/implementation/state.html
a
Ok, I'll try to test.Thanks
r
So, before the first request there is no
next_page_token
so it checks state for a
log_id
to start from.
a
what are you using log_expired for? can not understand
r
Auth0 does not retain logs indefinitely: https://auth0.com/docs/deploy-monitor/logs/log-data-retention
a
yeah, got it
@Reuben (Matatika) I hane one more question about replication_key_value In your example (oauth0), you got the value from the context_state.get("replication_key_value") field, but I do the same and for some reason I have another nested progress_markers object. What am I doing wrong?
I also don't understand how this post works for you. next(iter(extract_jsonpath("$[*].log_id", data)), None) If I'm not mistaken, in case your response will look something like this
Copy code
[
    {
       log_id: 1
       dateTime: "11-10-2022T12:03:00"
    },
    {
       log_id: 2
       dateTime: "11-10-2022T12:04:00"
    },
    {
       log_id: 3
       dateTime: "11-10-2022T12:06:00"
    },
]
Then this function will return 1. (although, in theory, get_next_page_token should have returned 3 in order to get all records where the id is greater than the last received in the next request)
r
In your example (oauth0), you got the value from the context_state.get("replication_key_value") field, but I do the same and for some reason I have another nested progress_markers object. What am I doing wrong?
Not sure. This is my state for
meltano run tap-auth0 target-jsonl
with only
stream_auth0_logs.*
selected:
Then this function will return 1. (although, in theory, get_next_page_token should have returned 3 in order to get all records where the id is greater than the last received in the next request)
This is just an Auth0-specific implementation. If there is no
next_page_token
or
log_id
state to start from, get the single oldest available log and begin pagination from there.
a
@Reuben (Matatika) I understood this, but I'm not talking about that. I mean why are you returning the id of the first entry in the array instead of the last? (I described a little more in a new post)