I am having an issue with `get replication key signpost` in Meltano #troubleshooting

I am having an issue with `get_replication_key_sig...

joshua_adeyemi

08/04/2021, 7:52 PM

I am having an issue with

get_replication_key_signpost

in the sdk, maybe I do not fully understand how that works. My understanding was that method is used to make sure that the replication key value that is stored is not above a certain value right? so for example, I overwrote the method to return

2021-08-04T19:08:35+00:00

(as a datetime object), but I see that the replication key value that is stored in my state is

2021-08-04T19:17:29Z

which is greater than the replication key signpost value. Am I misunderstanding what the behavior of that method should be?

joshua_adeyemi

08/04/2021, 7:53 PM

I tried with a much smaller timestamp even. Overwrote that

get_replication_key_signpost

to return

2021-08-02

(as a datetime object), but still getting the same results.

aaronsteers

08/04/2021, 7:58 PM

Hi, @joshua_adeyemi. This may be a bug. You are correct to expect the bookmark value to not pass the signpost value. To confirm though, there is a non-finalized state prior to stream completion, and there is a finalized/normal state. Can you provide the json state text to confirm? (Here or in an issue.)

aaronsteers

08/04/2021, 8:40 PM

I should add also, there may be sight differences in behavior depending upon whether the stream is sorted or not, whether it's partitioned, and whether there's a custom state granularity set.

joshua_adeyemi

08/04/2021, 9:36 PM

Where would you prefer? here? or I open an issue and the convo continues there?

joshua_adeyemi

08/04/2021, 9:37 PM

also to clarify, you want me to export the state file of the elt run correct?

joshua_adeyemi

08/04/2021, 10:10 PM

Here is an issue, with some sample data and statefile, and links to where in the code i think the issue might be occurring: https://gitlab.com/meltano/sdk/-/issues/180

3 Views

Open in Slack

Previous Next