How is the state stored in the postgresql backend? If we only embed the meltano.yml file in a docker container, will be able to reuse the state of previous executions?

We run meltano jobs in containers and pass

MELTANO_DATABASE_URI

as env var to container. meltano library picks it up and communicates with state db in the specified postrgesql uri

so as long as the tap and destination doesn't change, it would be able to recover the state?

If you use

run

then the tap and target names must remain the same for the same state to be used. If you use

elt

I believe you need to specify the same

--state-id

value as before.

I think that answers my question

You can always go to the state db and query the

state

and

run

tables to see what is the

state_id

that it has registered.

also check out https://docs.meltano.com/concepts/state_backends and https://docs.meltano.com/guide/integration#incremental-replication-state

I did a local test and so far so good, but I just realised that I can't use the same schema for multiple meltano projects because if they use the same tap and target, there would be a class - unless I add a custom suffix

You could use

inherit_from

in your meltano project

plugins:
  extractors:
    - name: my-named-source
      inherit_from: tap-postgres
      ...
  loaders:
    - name: my-named-destination
      inherit_from: target-redshift
      ...

then your `run`'s state will be created and maintained as

[
  {
    "state_id": "<environment>:my-named-source-my-named-destination",
    "partial_state": {},
    "completed_state": {..}
    "updated_at": "2023-06-29 05:05:26.689325"
  }
]

oh that's pretty neat actually. Thank you!