https://meltano.com/ logo
Channels
Powered by
#announcements
Title
# announcements
c

calm-hair-50010

03/01/2021, 5:37 PM
I’m running into an issue this morning in trying to take my local prototype and pack it up as a docker image and I’m curious if anyone has suggestions. I followed the instructions at https://meltano.com/docs/containerization.html#docker-compose which were straightforward and seemed to work just fine. When I go to run a pipeline using the new image though, I get an error that 
Config is missing required keys: ['user', 'password']
for tap-mysql. Those fields definitely exist within my 
meltano.yml
(which works fine running outside the container), and I’ve passed the credentials themselves in as environment variables during my 
docker run
. My only other guess is that perhaps meltano might be looking for a value saved in the SQLite DB that’s not exposed inside the container?
✅ 1
r

ripe-musician-59933

03/01/2021, 5:40 PM
@calm-hair-50010 If you use the new image to run 
meltano config tap-mysql
, do you see 
user
and 
password
there?
c

calm-hair-50010

03/01/2021, 5:41 PM
ah - surprisingly, I don’t
r

ripe-musician-59933

03/01/2021, 5:42 PM
Does 
meltano config tap-mysql list
show a value/source for those keys?
c

calm-hair-50010

03/01/2021, 5:42 PM
but I DO see other data returned by that command which defined in the same section of 
meltano.yml
as user/password, like host, database, etc.
r

ripe-musician-59933

03/01/2021, 5:42 PM
Is any of your own config getting picked up or only default values?
c

calm-hair-50010

03/01/2021, 5:43 PM
no, the host (for example) is what it should be based on my config
I think this is an environment variable issue - your second command shows that it knows those values should exist but they don’t have a value associated with them yet
r

ripe-musician-59933

03/01/2021, 5:44 PM
But you've set both user and password in 
meltano.yml
?
c

calm-hair-50010

03/01/2021, 5:45 PM
yes, but I’ve done that via environment variables - let me share a snippet
from `meltano.yml`:
Copy code
# PROD
      host: <redacted_but_a_static_value>
      user: $MYSQL_USER
      password: $MYSQL_PWD
then, the first chunk of my `docker run`:
Copy code
docker run \
    --volume $(pwd)/output:/project/output \
    --name meltano \
    -e MYSQL_USER=$MYSQL_USER \
    -e MYSQL_PWD=$MYSQL_PWD \
and then, results from `meltano config tap-mysql list`:
Copy code
user [env: TAP_MYSQL_USER] current value: None (from `meltano.yml`: '$MYSQL_USER')
password [env: TAP_MYSQL_PASSWORD] current value: None (from `meltano.yml`: '$MYSQL_PWD')
r

ripe-musician-59933

03/01/2021, 5:48 PM
Interesting, that suggests that 
MYSQL_USER
is not set when 
meltano config
is running. Do you see anything logged about that if you run with 
meltano --log-level=debug config
?
c

calm-hair-50010

03/01/2021, 5:50 PM
[2021-03-01 175028,342] [73|MainThread|meltano.core.utils] [DEBUG] Variable MYSQL_USER is missing from the environment. [2021-03-01 175028,342] [73|MainThread|meltano.core.utils] [DEBUG] Variable MYSQL_PWD is missing from the environment.
Maybe instead of trying to set those via the docker run, I should try setting them in the dockerfile itself? That’s something I haven’t yet tried
r

ripe-musician-59933

03/01/2021, 5:53 PM
docker run
should work. If you use 
-e MYSQL_USER=test
instead of 
-e MYSQL_USER=$MYSQL_USER
, does it work?
c

calm-hair-50010

03/01/2021, 5:54 PM
yep, seems to
(I’ve also confirmed that those env vars are set correctly in the terminal where I’m running 
docker run
)
r

ripe-musician-59933

03/01/2021, 5:57 PM
😕
I think you'll want to debug 
docker run -e
some more
c

calm-hair-50010

03/01/2021, 6:00 PM
yep, thanks for talking through it - I think you’re right!
r

ripe-musician-59933

03/01/2021, 6:01 PM
Good luck! 😄
c

calm-hair-50010

03/01/2021, 6:19 PM
well, for reasons I don’t really understand, “just trying the exact same thing again” seems to have solved the problem
I’ll take it!
👍 1
Powered by