https://linen.dev logo
Join Slack
Powered by
How’s everyone handling a custom webserver_config....
# docker
p
How’s everyone handling a custom webserver_config.py in airflow 2.0? What I did was adding the custom webserver_config.py in 
orchestrate/
directory and then adding below line to Dockerfile after 
COPY . .
, so that my custom webserver_config.py is copied over to 
.meltano/run/airflow
directory. 
RUN mkdir -p .meltano/run/airflow && cp -f orchestrate/webserver_config.py .meltano/run/airflow/webserver_config.py
I know that airflow 2.0 doesn’t yet support a custom location(or file) for WEBSERVER_CONFIG and it has hardcoded it to AIRFLOW_HOME + ‘/webserver_config.py’ , so probably it has to be fixed in airflow to read WEBSERVER_CONFIG through an environment variable before meltano can start supporting custom webserver_config.py. But meanwhile how are others handling this? What I am doing is working well for me, but I want to know if there is a better way to handle this?(Without modifying the default Dockerfile)
Probably a custom entrypoint.sh file would be nice to have, instead of entrypoint as 
["meltano"]
. I want to run couple of things before starting airflow-webserver and meltano UI. And all these can be combined into an entrypoint.sh file. The command in the above question can be added there. Additionally I also want to run 
meltano user add --overwrite <username> <password>
before starting the meltano ui (I am deploying everything via CICD and don’t want to create a user manually). I guess the custom entrypoint.sh would address this. Any thoughts?
d
@pankaj_saini I don't think there's a better alternative to your webserver_config workaround right now, until a setting is added to Airflow. Meltano could support 
orchestrate/webserver_config.py
out of the box and handle copying it itself, but I'd rather wait for the new setting.
A custom entrypoint also makes sense if you want to do some additional preparation that isn't appropriate (or won't work) in the Dockerfile. The Dockerfile Meltano provides (and everything else in your project directory) is meant as a starting point for you to edit as appropriate for your circumstances, so what you're doing is totally valid 🙂
p
Thanks @douwe_maan, this is really helpful.
m
Seems the issue filed for Airflow is pretty stale https://github.com/apache/airflow/issues/10206. Is there any chance Meltano is going to bake this in?
d
@maarten_van_gijssel I think this could be added to Meltano, but we don’t have an issue yet! Can you please file one since you have a use case for this?
m
d
cc @taylor since this is a feature proposal!
n
Just putting a +1 on this 🙂
Open in Slack
PreviousNext
Powered by