I’m running into an issue with docker (full disclosure, I might be missing something pretty fundamental here). When I build the docker container I would like to include

meltano install

in the build so I don’t have to rebuild it every time I run a

meltano

command (we don’t have ephemeral storage on production). This doesn’t work though because the

Dockerfile

doesn’t have access to the

MELTANO_DATABASE_URI

(and can’t because of the way we handle secrets) and the env variable is required to run

meltano install

. I guess my main question is: why does meltano need access to the Metadata Database to run

meltano install

? I don’t see anything in the db that relates to installing the plugins.

When I build the docker container I would like to include 

meltano install

 in the build so I don’t have to rebuild it every time I run a 

meltano

 command

@matt_cooley Makes sense, that's what the recommended Dockerfile from https://meltano.com/docs/containerization.html does as well: https://gitlab.com/meltano/files-docker/-/blob/master/bundle/Dockerfile#L12

This doesn’t work though because the 

Dockerfile

 doesn’t have access to the 

MELTANO_DATABASE_URI

 (and can’t because of the way we handle secrets) and the env variable is required to run 

meltano install

.

How do you mean, "this doesn't work"? Are you seeing an error message?

MELTANO_DATABASE_URI

shouldn't be required since the

database_uri

setting has a default value: https://meltano.com/docs/settings.html#database-uri

why does meltano need access to the Metadata Database to run 

meltano install

?

Installation doesn't actually need access to the system database, but most

meltano

commands connect with the system database and run migrations just to make sure its table structure is compatible with the model classes living in the codebase, that might be used as part of the command. This unnecessary system database requirement shouldn't cause any issues, though, since it should just use the local system database at

.meltano/meltano.db

if no remote database URI is configured. If it's causing you trouble, that's a bug 🙂

Here’s what I found: If the

.meltano

folder is already built (using docker-compose, so it has access to the meltano postgres db I’m using). Then running

meltano install

in the Docker container works just fine. If I delete the

.meltano

folder first (which mimics a deploy env) then I get this error:

+ meltano install
Can't initialize database: (sqlite3.OperationalError) unable to open database file
(Background on this error at: <http://sqlalche.me/e/13/e3q8>)

A curious note though. Regardless whether it fails or not I’m not seeing the

meltano.db

file. Which makes sense for the first situation, but maybe it doesn’t for the second one?

@matt_cooley OK! We should make sure then that

.meltano

is created automatically before a connection to the system database is attempted. Can you please create an issue for that? Should be really easy to fix, you may even be able to contribute it 😉

Cool. Will do. I’m a bit confused why the `check_db_connection` function isn’t catching this, assuming I’m reading it write and the

project_engine

decorator is where this error is coming from. Anyway, I’ll get a ticket up either way. Thanks!

@matt_cooley I think it's failing already in

create_engine

, before it ever gets to

check_db_connection

.

project_engine

is called from the

pass_project

decorator defined here: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/cli/params.py#L50, which is used here: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/cli/install.py#L18

Yeah, that’s what I was trying to say.

pass_project

calls

project_engine

, which then runs

check_db_connection

(which I would expect to fail, but the error message I get

Can't initialize database: (sqlite3.OperationalError) unable to open database file

appears to be from

inti_hook

which is called after

check_db_connection

Ah, that's interesting

I would expect the error to happen here

Right... I think it's worth figuring out why that's not happening. Perhaps

except OperationalError:

is not catching

sqlite3.OperationalError

?

It catches it when I did a quick test locally. Yeah, I dunno

Thanks for creating the issue, I'll check it out tomorrow. Should be as easy as adding an mkdir call somewhere, as you suggested. I'm not sure why it's failing in your Airflow docker image while it's fine here: https://gitlab.com/meltano/demo-project/-/jobs/943770251#L283, where we use

meltano/meltano

(https://gitlab.com/meltano/demo-project/-/blob/master/Dockerfile)

Yeah I dunno. Stumped me and my boss

@matt_cooley I've responded: https://gitlab.com/meltano/meltano/-/issues/2498#note_479751403

Rad! I’m a little swamped right now, but I’ll definitely see if I can take a crack at that when I have the time. Thanks!