today I set up a meltano pipeline in azure app container job with state backend on a storage container. the container is invoked with the following command

meltano el --environment=test tap-snowflake target-db2

this results in state files with the following format & each time the meltano job is launched, a new state file is written, where the timestamp at the front corresponds to when the job was launched:

yyyy-mm-ddThhmmss--tap-snowflake--target-db2/state.json

and inspecting the logs, it seems each on each run, meltano is looking for a statefile with the launch time, eg.

No state found for 2024-09-11T155644--tap-snowflake--target-db2

To get around this, I've updated the container command to

meltano el --environment=test tap-snowflake target-db2 --state-id=tap-snowflake--target-db2

However, I am wondering if there is a more elegant way to specify the state-id. Also, does this feel like a bug or is this expected but not clearly documented? when running locally, the state is saved to a location that includes the environment, tap and target names, e.g.

dev:tap-snowflake-to-target-db2/state.json

The

--state-id

(prev.

--job_id

) argument is as old as Meltano and IIUC it's there to make incremental replication an opt-in behavior.

However, I am wondering if there is a more elegant way to specify the state-id.

There's the

MELTANO_STATE_ID

env var, but that may also be inelegant.

Also, does this feel like a bug or is this expected but not clearly documented?

It's expected. There's documentation: • https://docs.meltano.com/guide/troubleshooting/#incremental-replication-not-running-as-expected • https://docs.meltano.com/guide/integration/#incremental-replication-state Though of course documentation amendments and additions in places where they make the most sense for a newcomer are always welcome 😃.

when running locally, the state is saved to a location that includes the environment, tap and target names, e.g.

Copy code

dev:tap-snowflake-to-target-db2/state.json

I may be wrong, but this seems to be using

meltano run

instead?

Yes, it seems like its documented & you're right I guess I was using meltano run locally.