I'm trying to figure out how to manually set state and replication key value. I've tried using the

finalize_state_progress_markers

function, but it seems like no matter what format I try, the state isn't persisted when I check it (either from the command line with

meltano state get <my_elt_pipeline_job>

, or with the

stream_state

and

get_context_state

functions). Here's some additional context about the problem: https://meltano.slack.com/archives/C01PKLU5D1R/p1682455126687399?thread_ts=1681998358.444049&cid=C01PKLU5D1R

FYI @edgar_ramirez_mondragon and I were just chatting about this. It isnt forgotten 😅 , were still looking into the best way to do it

haha thanks Pat! And a quick update to correct some of my prior ignorance:

finalize_state_progress_markers

is working to set state. I was either setting the state incorrectly in that function, or reading the state value from the wrong meltano project (🤦 ) but either way, I'm seeing it set now!

@omar_abed oh great! So does that mean your backfill problem is solved? Or what pending challenges are you still working through

Sorry, another correction... I noticed the state was only getting set after a call to

get_records

completed, but not each time

finalize_state_progress_markers

is called. So maybe

finalize_state_progress_markers

isn't setting state at all, like I originally thought. I need to dive into that function a bit to see what it's really doing.