anyone aware of how to get Meltano and Dagster to share the same postgres database for their logging? I’m just trying to get a simple docker-compose version of this working locally but it seems that setting things up for dagster will break things for meltano and and vice versa.

Hi, @josh_lloyd. Do you mean the database and schema that meltano and airflow use internally to store their data in Postgres? You should be able to override the default DB and schema if needed if there are collisions in those namespaces between each app’s storage in postgres.

meltano and dagster, but yes. I can get them both pointed at the same postgres database but when meltano goes to create the alembic table (and probably most of the other tables) it chokes saying that there is schema name provided for alembic table. I don’t know if there is anyway to provide that to meltano when it tries to create all the tables …

I remember someone else overriding the db and/or schema that meltano uses in postgres - and also the schema used for Airflow, but I didn’t find them in the docs after a quick search. Any reason not to just use distinct databases in postgres? (Pinging @douwe_maan also in case there’s a docs ref or some other trick I’m missing.)

only because I was trying to avoid paying for two postgres databases in the cloud once I deploy it. A kind of petty reason since I’m sure they’ll be pretty small databases

Oh - that makes sense. I wasn’t considering a per-db cost for the same reason.

oh, shoot I was using that wrong …

Great - thanks.

YOu can have two databases in the same postgres RDS instance in AWS

yes, to be clear, in order to enable this I had to create a separate schema in the default database and made sure that I added the option to the

MELTANO_DATABASE_URI

that AJ mentioned above and pointed it to that newly created schema. As soon as I got that setup right. It worked. Thanks to all for the help!!

@chris_kings-lynne You are absolutely right. Sorry, I didn’t mean to suggest you couldn’t have two separate databases. I think the only reason I didn’t do that was more of an accident than anything. I kept getting errors when I tried to do that and it took me a while to realize that I had to

docker volume remove meltano_postgresql_data

every time I wanted to redo the db container from scratch, otherwise it uses the credentials you set the last time your ran

docker-compose up

. But by the time I figured that out, I had already moved into a configuration where I had the dagster tables and meltano tables in two separate schemas within the same default database.

No probs