I'm creating a new tap at <https://github.com/ilkk...
# singer-tap-development
I'm creating a new tap at https://github.com/ilkkapeltola/tap-sirene/ What this tap does is it queries the public French company database and fetches all company data they have published. But I have challenges using state and resuming from where I left off. I get these kind of info statements in my output
Copy code
2022-06-01T11:41:35.176643Z [info     ] time=2022-06-01 11:41:35 name=target_snowflake level=INFO message=Emitting state {"bookmarks": {"siren": {"replication_key_signpost": "2022-06-01T11:12:43.534679+00:00", "starting_replication_value": "2000-01-01T00:00:01", "progress_markers": {"Note": "Progress is not resumable if interrupted.", "replication_key": "dateDernierTraitementUniteLegale", "replication_key_value": "2008-09-20T04:50:47"}}}} cmd_type=loader job_id=sirene-prod-1 name=target-sf-transferwise run_id=29bdbab7-15b6-4335-9d47-3c5f170904ce stdio=stderr
So it is storing the progress marker, but why does it say as a note "Progress is not resumable if interrupted"? What do I need to change for it to be able to resume? The API itself is a little quirky, I'll describe what is happening in my get_url_params in thread, since I believe it could be something to do with that.