Guys Do I need to setup external MELTANO DATABASE URI if I a

Question

Guys Do I need to setup external MELTANO DATABASE URI if I am running meltano in Gitlab pipeline SQLite db is deleted after each run Btw I tried to use Azure Postgresql ` base etl warehouse postgresql 5432 meltano sslmode=require` but it does not work due to ` psycopg2 OperationalError could not translate host name to address Name or service not known` which is nonsense because I am using same server different db for target postgres note that Azure is using `sysadmin base etl warehouse postgresql` format of username

ripe-musician-59933 · Answer

< bumpy ram 82773> If you re running in a GitLab pipeline using an external system database is definitely easiest although you could also use CI artifacts to store ` meltano meltano db` or just the state file Your Azure PostgresQL URL is likely failing because of that ` ` in the username which is interpreted as the username host separator If you encode it as ` 40` int the URL it should make it to Azure as ` ` ``````

bumpy-ram-82773 · Answer

< ripe musician 59933> Thanks Is OK to have single server but different databases for ETL and for Meltano itself

ripe-musician-59933 · Answer

< bumpy ram 82773> Yep that s fine

bumpy-ram-82773 · Answer

Seems that encoding ` ` char does not work Did You ever try to use Azure Postgresql

ripe-musician-59933 · Answer

Something else to check Is the hostname `` correct or should it be `` with an extra ` `

ripe-musician-59933 · Answer

I don t have experience with Azure PostgreSQL specifically but I think what s more relevant here is how SQLAlchemy interprets database URLs <https docs sqlalchemy org en 14 core engines html database urls>

bumpy-ram-82773 · Answer

Oh You are right I did not notice Thanks No need to escape ` `

bumpy-ram-82773 · Answer

I tried also with caching state file but meltano will not run without state file Its bit problematic but external database is ok for us

ripe-musician-59933 · Answer

gt meltano will not run without state file It should be able to Did you see an error

bumpy-ram-82773 · Answer

I know why cuz meltano looks for that file relatively to project directory

ripe-musician-59933 · Answer

< bumpy ram 82773> Ah if you specify ` state lt path gt ` on the CLI or `state lt path gt ` in `meltano yml` Meltano will always expect that file to exist Do you have a way of not passing ` state` when you don t have one yet Alternatively you can just write ` ` to a file to serve as an empty state file