matt_cooley
01/04/2021, 11:06 PMmeltano install
in the build so I don’t have to rebuild it every time I run a meltano
command (we don’t have ephemeral storage on production).
This doesn’t work though because the Dockerfile
doesn’t have access to the MELTANO_DATABASE_URI
(and can’t because of the way we handle secrets) and the env variable is required to run meltano install
.
I guess my main question is: why does meltano need access to the Metadata Database to run meltano install
? I don’t see anything in the db that relates to installing the plugins.douwe_maan
01/04/2021, 11:36 PMWhen I build the docker container I would like to include@matt_cooley Makes sense, that's what the recommended Dockerfile from https://meltano.com/docs/containerization.html does as well: https://gitlab.com/meltano/files-docker/-/blob/master/bundle/Dockerfile#L12in the build so I don’t have to rebuild it every time I run ameltano install
commandmeltano
This doesn’t work though because theHow do you mean, "this doesn't work"? Are you seeing an error message?doesn’t have access to theDockerfile
(and can’t because of the way we handle secrets) and the env variable is required to runMELTANO_DATABASE_URI
.meltano install
MELTANO_DATABASE_URI
shouldn't be required since the database_uri
setting has a default value: https://meltano.com/docs/settings.html#database-uri
why does meltano need access to the Metadata Database to runInstallation doesn't actually need access to the system database, but most?meltano install
meltano
commands connect with the system database and run migrations just to make sure its table structure is compatible with the model classes living in the codebase, that might be used as part of the command.
This unnecessary system database requirement shouldn't cause any issues, though, since it should just use the local system database at .meltano/meltano.db
if no remote database URI is configured.
If it's causing you trouble, that's a bug 🙂matt_cooley
01/05/2021, 12:07 AM.meltano
folder is already built (using docker-compose, so it has access to the meltano postgres db I’m using). Then running meltano install
in the Docker container works just fine.
If I delete the .meltano
folder first (which mimics a deploy env) then I get this error:
+ meltano install
Can't initialize database: (sqlite3.OperationalError) unable to open database file
(Background on this error at: <http://sqlalche.me/e/13/e3q8>)
A curious note though. Regardless whether it fails or not I’m not seeing the meltano.db
file. Which makes sense for the first situation, but maybe it doesn’t for the second one?matt_cooley
01/05/2021, 12:59 AMmkdir .meltano
before meltano install
. That said, it’s still not working. I think this is some Docker magic that I’m not understanding.matt_cooley
01/05/2021, 5:11 AM.meltano
directory.douwe_maan
01/05/2021, 4:07 PM.meltano
is created automatically before a connection to the system database is attempted. Can you please create an issue for that? Should be really easy to fix, you may even be able to contribute it 😉matt_cooley
01/05/2021, 5:38 PMproject_engine
decorator is where this error is coming from. Anyway, I’ll get a ticket up either way. Thanks!douwe_maan
01/05/2021, 5:40 PMcreate_engine
, before it ever gets to check_db_connection
. project_engine
is called from the pass_project
decorator defined here: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/cli/params.py#L50, which is used here: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/cli/install.py#L18matt_cooley
01/05/2021, 5:45 PMpass_project
calls project_engine
, which then runs check_db_connection
(which I would expect to fail, but the error message I get Can't initialize database: (sqlite3.OperationalError) unable to open database file
appears to be from inti_hook
which is called after check_db_connection
douwe_maan
01/05/2021, 5:45 PMmatt_cooley
01/05/2021, 5:46 PMdouwe_maan
01/05/2021, 5:47 PMexcept OperationalError:
is not catching sqlite3.OperationalError
?matt_cooley
01/05/2021, 5:52 PMmatt_cooley
01/05/2021, 8:10 PM.meltano
directory if it doesn’t exist), but fails in the airflow docker image.matt_cooley
01/05/2021, 9:01 PM1.62.0
which is why the exceptions was throwing where it wasdouwe_maan
01/05/2021, 9:03 PMmeltano/meltano
(https://gitlab.com/meltano/demo-project/-/blob/master/Dockerfile)matt_cooley
01/05/2021, 10:57 PMdouwe_maan
01/06/2021, 6:47 PMmatt_cooley
01/06/2021, 8:30 PM