Hi we have an existing meltano instance on version

v1.105.0

that we're preparing to migrate to

v3.0.0

for. v1 is currently using a postgres database as its state backend. I've gone ahead and just created a new project running v3, rather than modifying the existing one. My question is would we be able to simply point the new project that's running v3 to the same backend uri while maintaining the existing bookmarks? i've been running v3 locally and inspecting the tables using the local file system in

.meltano/meltano.db

vs the tables in our postgres instance and it looks like v3 has some new job related tables

state

and

runs

vs v1 which only has the

job

table. Would someone be able to give me more info on how state was maintained in v1 vs any differences in v3?

Some tables were renamed in v2. The safest way to update the database is to run

meltano config meltano list

with v3, and the database uri still pointing to the original database, to make sure schema migrations run before you run any pipeline.

thanks @edgar_ramirez_mondragon would you be able to give an example of how we would set the backend uri if we have multiple environments with differing state backends? I see both

MELTANO_DATABASE_URI

and

MELTANO_STATE_BACKEND_URI

env vars, what is the difference between the two? if we want dev to point to local file system, but prod to postgres endpoint how would the

meltano.yml

look like?

what is the difference between the two?

The database URI is used regardless of the state backend you configure. By defaults points to a sqlite db in

.meltano/meltano.db

. This is not critical for the operation of Meltano if you're using a different state backend.

if we want dev to point to local file system, but prod to postgres endpoint how would the

meltano.yml

look like?

I would instead use env vars: • Local:

MELTANO_STATE_BACKEND_URI='file:///${MELTANO_SYS_DIR_ROOT}/state'

• Prod:

MELTANO_DATABASE_URI='postgresql+psycopg://<username>:<password>@<host>:<port>/<database>'

https://docs.meltano.com/concepts/state_backends/#local-file-system https://docs.meltano.com/reference/settings?meltano-tabs=env

is running

meltano config meltano list

with v3 after pointing to the existing v1 backend effectively the same as running

meltano upgrade database

or would i still have to run that command afterwards

The former should be enough to trigger db migrations

@edgar_ramirez_mondragon thanks for the clarification! Are there any rollbacks that we'd be able to do if the schema migration doesn't go as planned? after the update would the old v1 meltano pipelines still be able to run if the db is using v3 schema? or do we need a fast follow with upgrading meltano to v3?

Are there any rollbacks that we'd be able to do if the schema migration doesn't go as planned?

In theory, yes. Meltano uses alembic, so you could downgrade the db schema using the alembic CLI installed in the same environment as Meltano to revert to revision

13e8639c6d2b

.

after the update would the old v1 meltano pipelines still be able to run if the db is using v3 schema? or do we need a fast follow with upgrading meltano to v3?

No. Pipelines running on Meltano v1 will not work on v3 because things are renamed and moved under the hood in the db schema.

sounds good, thanks so much for your help!