https://linen.dev logo
Join Slack
Powered by
I’m running into an issue this morning in trying t...
# announcements
n
I’m running into an issue this morning in trying to take my local prototype and pack it up as a docker image and I’m curious if anyone has suggestions. I followed the instructions at https://meltano.com/docs/containerization.html#docker-compose which were straightforward and seemed to work just fine. When I go to run a pipeline using the new image though, I get an error that 
Config is missing required keys: ['user', 'password']
for tap-mysql. Those fields definitely exist within my 
meltano.yml
(which works fine running outside the container), and I’ve passed the credentials themselves in as environment variables during my 
docker run
. My only other guess is that perhaps meltano might be looking for a value saved in the SQLite DB that’s not exposed inside the container?
d
@nick_hamlin If you use the new image to run 
meltano config tap-mysql
, do you see 
user
and 
password
there?
n
ah - surprisingly, I don’t
d
Does 
meltano config tap-mysql list
show a value/source for those keys?
n
but I DO see other data returned by that command which defined in the same section of 
meltano.yml
as user/password, like host, database, etc.
d
Is any of your own config getting picked up or only default values?
n
no, the host (for example) is what it should be based on my config
I think this is an environment variable issue - your second command shows that it knows those values should exist but they don’t have a value associated with them yet
d
But you've set both user and password in 
meltano.yml
?
n
yes, but I’ve done that via environment variables - let me share a snippet
from `meltano.yml`:
Copy code
# PROD
      host: <redacted_but_a_static_value>
      user: $MYSQL_USER
      password: $MYSQL_PWD
then, the first chunk of my `docker run`:
Copy code
docker run \
    --volume $(pwd)/output:/project/output \
    --name meltano \
    -e MYSQL_USER=$MYSQL_USER \
    -e MYSQL_PWD=$MYSQL_PWD \
and then, results from `meltano config tap-mysql list`:
Copy code
user [env: TAP_MYSQL_USER] current value: None (from `meltano.yml`: '$MYSQL_USER')
password [env: TAP_MYSQL_PASSWORD] current value: None (from `meltano.yml`: '$MYSQL_PWD')
d
Interesting, that suggests that 
MYSQL_USER
is not set when 
meltano config
is running. Do you see anything logged about that if you run with 
meltano --log-level=debug config
?
n
[2021-03-01 175028,342] [73|MainThread|meltano.core.utils] [DEBUG] Variable MYSQL_USER is missing from the environment. [2021-03-01 175028,342] [73|MainThread|meltano.core.utils] [DEBUG] Variable MYSQL_PWD is missing from the environment.
Maybe instead of trying to set those via the docker run, I should try setting them in the dockerfile itself? That’s something I haven’t yet tried
d
docker run
should work. If you use 
-e MYSQL_USER=test
instead of 
-e MYSQL_USER=$MYSQL_USER
, does it work?
n
yep, seems to
(I’ve also confirmed that those env vars are set correctly in the terminal where I’m running 
docker run
)
d
😕
I think you'll want to debug 
docker run -e
some more
n
yep, thanks for talking through it - I think you’re right!
d
Good luck! 😄
n
well, for reasons I don’t really understand, “just trying the exact same thing again” seems to have solved the problem
I’ll take it!
11 Views
Open in Slack
PreviousNext
Powered by