I’m running into an issue with docker (full disclo...
# plugins-general
m
I’m running into an issue with docker (full disclosure, I might be missing something pretty fundamental here). When I build the docker container I would like to include
meltano install
in the build so I don’t have to rebuild it every time I run a
meltano
command (we don’t have ephemeral storage on production). This doesn’t work though because the
Dockerfile
doesn’t have access to the
MELTANO_DATABASE_URI
(and can’t because of the way we handle secrets) and the env variable is required to run
meltano install
. I guess my main question is: why does meltano need access to the Metadata Database to run
meltano install
? I don’t see anything in the db that relates to installing the plugins.
d
When I build the docker container I would like to include 
meltano install
 in the build so I don’t have to rebuild it every time I run a 
meltano
 command
@matt_cooley Makes sense, that's what the recommended Dockerfile from https://meltano.com/docs/containerization.html does as well: https://gitlab.com/meltano/files-docker/-/blob/master/bundle/Dockerfile#L12
This doesn’t work though because the 
Dockerfile
 doesn’t have access to the 
MELTANO_DATABASE_URI
 (and can’t because of the way we handle secrets) and the env variable is required to run 
meltano install
.
How do you mean, "this doesn't work"? Are you seeing an error message?
MELTANO_DATABASE_URI
shouldn't be required since the
database_uri
setting has a default value: https://meltano.com/docs/settings.html#database-uri
why does meltano need access to the Metadata Database to run 
meltano install
?
Installation doesn't actually need access to the system database, but most
meltano
commands connect with the system database and run migrations just to make sure its table structure is compatible with the model classes living in the codebase, that might be used as part of the command. This unnecessary system database requirement shouldn't cause any issues, though, since it should just use the local system database at
.meltano/meltano.db
if no remote database URI is configured. If it's causing you trouble, that's a bug 🙂
m
Here’s what I found: If the
.meltano
folder is already built (using docker-compose, so it has access to the meltano postgres db I’m using). Then running
meltano install
in the Docker container works just fine. If I delete the
.meltano
folder first (which mimics a deploy env) then I get this error:
Copy code
+ meltano install
Can't initialize database: (sqlite3.OperationalError) unable to open database file
(Background on this error at: <http://sqlalche.me/e/13/e3q8>)
A curious note though. Regardless whether it fails or not I’m not seeing the
meltano.db
file. Which makes sense for the first situation, but maybe it doesn’t for the second one?
Ok, I got it to run by adding a
mkdir .meltano
before
meltano install
. That said, it’s still not working. I think this is some Docker magic that I’m not understanding.
Ok, so it was some docker magic that I wasn’t understanding. It does seem to be failing because there is no
.meltano
directory.
d
@matt_cooley OK! We should make sure then that
.meltano
is created automatically before a connection to the system database is attempted. Can you please create an issue for that? Should be really easy to fix, you may even be able to contribute it 😉
m
Cool. Will do. I’m a bit confused why the `check_db_connection` function isn’t catching this, assuming I’m reading it write and the
project_engine
decorator is where this error is coming from. Anyway, I’ll get a ticket up either way. Thanks!
d
@matt_cooley I think it's failing already in
create_engine
, before it ever gets to
check_db_connection
.
project_engine
is called from the
pass_project
decorator defined here: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/cli/params.py#L50, which is used here: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/cli/install.py#L18
m
Yeah, that’s what I was trying to say.
pass_project
calls
project_engine
, which then runs
check_db_connection
(which I would expect to fail, but the error message I get
Can't initialize database: (sqlite3.OperationalError) unable to open database file
appears to be from
inti_hook
which is called after
check_db_connection
d
Ah, that's interesting
m
I would expect the error to happen here
d
Right... I think it's worth figuring out why that's not happening. Perhaps
except OperationalError:
is not catching
sqlite3.OperationalError
?
m
It catches it when I did a quick test locally. Yeah, I dunno
Added an issue. I added some additional notes. Turns out it will work fine on my Mac (creates the
.meltano
directory if it doesn’t exist), but fails in the airflow docker image.
Also, I’m on
1.62.0
which is why the exceptions was throwing where it was
d
Thanks for creating the issue, I'll check it out tomorrow. Should be as easy as adding an mkdir call somewhere, as you suggested. I'm not sure why it's failing in your Airflow docker image while it's fine here: https://gitlab.com/meltano/demo-project/-/jobs/943770251#L283, where we use
meltano/meltano
(https://gitlab.com/meltano/demo-project/-/blob/master/Dockerfile)
m
Yeah I dunno. Stumped me and my boss
d
m
Rad! I’m a little swamped right now, but I’ll definitely see if I can take a crack at that when I have the time. Thanks!