is it a bad practice to set the meltano system db to the postgres db that’s being used for ELT ?
a
aaronsteers
12/05/2022, 11:48 PM
Probably this will be fine if your data is at relatively small volumes. In that case, you'll want to make sure each is landing in separate schemas/dbs in postgres
c
christoph
12/06/2022, 12:17 AM
I normally would use the Datawarehouse DB strictly for actual business data and anything else like dbt utility tables.
And then the "metadata" dbs for tooling like Metlano, Superset, Airflow/Dagster would each have their own DBs
Separating out the different tables in the same DB using schemas / namespaces is another possibility.
I think the 2 main concerns for designing are backup/restore and security access models.
m
Matt Menzenski
12/06/2022, 4:18 PM
Thanks!
Our data platform is three parts, an S3 data lake, a Postgres RDS operational data store, and then a Redshift dimensional warehouse. So we’d be potentially using that ODS database here, not the actual warehouse.
c
christoph
12/06/2022, 8:16 PM
Oh, right. Yeah. Using the same Posgtres RDS Cluster (which AWS calls a "DB instance") to host your transactional business data and also your Meltano metadata in different databases on that same Postgres cluster sounds fine.