I just switched from the PostgreSQL state backend ...
# troubleshootingj
jan_soubusta
03/01/2023, 7:36 PMI just switched from the PostgreSQL state backend to AWS S3.
I realized that for the local development, I would appreciate connecting to local Minio instead of forcing developers to have RW access to AWS S3.
Anyone knows how to set state_backend.uri to point to Minio?
jan_soubusta
03/01/2023, 7:49 PMAlternative - use SQLLite state backend while developing locally.
Is it possible to turn on AWS S3 backend only when not running locally?
I don't see any other option than to duplicate meltano.yml for local development, which is not convenient...
e
edgar_ramirez_mondragon
03/01/2023, 7:50 PMYou could use environment variables that are set only in production to enable the S3 backend. There’s a nice example in https://github.com/meltano/meltano/blob/b1f65b713a2a1975655a7ee3f345bbf180455288/docs/example-library/meltano-state-s3/index.md
j
jan_soubusta
03/01/2023, 8:02 PMendpoint_url it is! Thanks!
m
Matt Menzenski
03/02/2023, 1:58 PMWe set the meltano system database URI environment variable in our remote jobs. Developers running things locally use the default sqllite db.
j
jan_soubusta
03/02/2023, 2:00 PM@Matt Menzenski works as well.
But I prefer testing as much as possible locally, so I added Minio to the docker-compose used for local development/testing and am overriding only credentials/endpoint_url.