https://linen.dev logo
Join Slack
Powered by
Hi there. Recently tried deploying a newer version...
# troubleshooting
c
Hi there. Recently tried deploying a newer version of Meltano (1.83) with Airflow 2.1.2 (AF2) and I'm running into snags, mostly related to AF2 configuration that I had not foreseen. Initially, this was a red herring that was related to me not dropping the airflow database and starting fresh. But now that no longer seems to be the case.
For example. AF2 now seems to require AuthN. I tried disabling this (we're behind an OAuth-based system in GCP) but that didn't work, so I'm now creating a user, programmatically, in the airflow database.
Another thing I've noticed is that, whereas before, Task Instance logs were accessible via a link in the UI, now they are incorrectly using a 
localhost
base URL. Googling for why, I'm advised to set an env var or config var that indicates the base URL: https://stackoverflow.com/a/49812929/148680
With Airflow 1 and previous versions of Meltano, much of the configuration (aside from, say, 
pip_url
) was taken care of. However, this coupling of versions is requiring a much firmer grasp of Airflow's configuration. But no one else seems to be having these same issues so I think I'm doing something that is against common practice.
e
Hi @casey! It might be the case that some settings new in 2.0 were not integrated into the plugin definition. (It might make sense to add 
AIRFLOW__WEBSERVER__BASE_URL
and have it default to 
MELTANO_UI_BIND_HOST
). Is that the case, are you deploying Airflow and Meltano under the same base url?
I see. That setting is not currently "known" to Meltano and as a temporary workaround you could set 
AIRFLOW__WEBSERVER__BASE_URL
directly in the server/container Meltano is running in.
I also created an issue.
c
Hi @edgar_ramirez_mondragon. Thank you for very gracously taking the time to reply and file an issue on my behalf. I wanted to do a little more testing so that I could offer up substantive information regarding this specific issue in response. I had been under the impression that configuration specified in the Airflow section of orchestrators was simply passed on to Airflow... I didn't realize that the discovery process constrained configuration to a subset of potential key/value pairs. That explains why my attempts to set 
webserver.base_url
failed. As a workaround (which I'll add to the issue you created) I set 
AIRFLOW__WEBSERVER__BASE_URL
in the shell that runs 
meltano invoke airflow webserver
. Regarding the other issue of Airflow 2 AuthN: it seems that one must manipulate 
$AIRFLOW_HOME/webserer_config.py.
As it stands, a default file is put in place with the line 
AUTH_DB
uncommented. It would be nice to avoid keeping this file in version control and/or manipulating it in the container. There is an issue for this here: https://gitlab.com/meltano/meltano/-/issues/2909 and here are the relevant Airflow docs: http://airflow.apache.org/docs/apache-airflow/2.1.2/security/webserver.html#web-authentication
e
Ah thanks for commenting on the issue and @casey simple smile. And the 
webserver_config.py
seems like something we can figure out with a short turnaround
Open in Slack
PreviousNext
Powered by