https://linen.dev logo
Join Slack
Powered by
Has anyone managed to get the airflow UI working b...
# announcements
n
Has anyone managed to get the airflow UI working behind a reverse proxy? eg / goes to meltano, /airflow/ goes to airflow. I'm using nginx with the following config:
Copy code
location /airflow/ {	
                proxy_pass <http://airflow-ui:8080/airflow/>;	
                proxy_set_header Host $host;	
                proxy_redirect off;	
                proxy_http_version 1.1;	
                proxy_set_header Upgrade $http_upgrade;	
                proxy_set_header Connection "upgrade";	
            }
But when I go to /airflow/ I just get a 
Internal Server Error
. When I go to the logs of the airflow container, I can see:
Copy code
[2020-09-22 07:49:20 +0000] [154] [ERROR] Error handling request /airflow/
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/sync.py", line 135, in handle
    self.handle_request(listener, req, client, addr)
  File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/sync.py", line 176, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/usr/local/lib/python3.7/site-packages/werkzeug/middleware/dispatcher.py", line 66, in __call__
    return app(environ, start_response)
  File "/code/.meltano/orchestrators/airflow/venv/lib/python3.7/site-packages/airflow/www/app.py", line 172, in root_app
    resp(b'404 Not Found', [(b'Content-Type', b'text/plain')])
  File "/usr/local/lib/python3.7/site-packages/gunicorn/http/wsgi.py", line 253, in start_response
    self.process_headers(headers)
  File "/usr/local/lib/python3.7/site-packages/gunicorn/http/wsgi.py", line 260, in process_headers
    raise TypeError('%r is not a string' % name)
TypeError: b'Content-Type' is not a string
10.2.97.37 - - [22/Sep/2020:07:49:20 +0000] "GET /airflow/ HTTP/1.1" 500 0 "-" "-"
AIRFLOW__WEBSERVER__BASE_URL
is set with {host_name}/airflow and 
AIRFLOW__WEBSERVER__ENABLE_PROXY_FIX
is set to 'true'.
a
What's the value for 
base_url
in your 
.meltano/orchestrators/airflow/airflow.cfg
file? [EDIT- just saw you've set that correctly) Have you tried adding 
/airflow/
to the end of 
proxy_pass
in your nginx conf?
n
Hey @al_whatmough thanks for helping out - no joy though sadly.
a
@niall_woodward ah thought it could be too good to be true if such a simple change would fix it!
n
You're right to check though it's often always the simple things that break this sort of stuff!
a
Are you running the default airflow instance bundled with Airflow? Or have you set up a separate airflow container and are pointing Meltano's dag generator at that instance?
I'm actually not on my laptop right now so finding it hard to check which version of Airlow Meltano uses. That PR you linked to looks relevant, I notice it was merged a while ago though
Either way, to me it looks like the underlying issue is that it's actually returning a 404 🧐
n
I'm using the built in airflow meltano. Started with 
"meltano", "invoke", "airflow", "webserver"
And I'm adding the required files in the dockerfile with 
RUN meltano add files airflow
All the env vars for the airflow UI container:
Copy code
- name: "MELTANO_DATABASE_URI"
          value: {{ .Values.database_url }}/meltano
        - name: "AIRFLOW__CORE__SQL_ALCHEMY_CONN"
          value: {{ .Values.database_url }}/airflow
        - name: "MELTANO_READONLY"
          value: "1"
        - name: "AIRFLOW__WEBSERVER__BASE_URL"
          value: {{ .Values.hostname }}/airflow
        - name: "AIRFLOW__WEBSERVER__ENABLE_PROXY_FIX"
          value: "true"
a
Looks fine to me. Can't see any obvious issues. If I can't help here hopefully someone else can step in! The only other thing I'd check if you haven't already is whether removing the trailing / in the nginx conf makes a difference. Or from memory if you do location ^~ /airflow that should do a regex match
n
From my understanding the trailing slash is what tells nginx to remove the location specifier from the URL, so without it the airflow service would receive /airflow/airflow
Think there are probably two issues here, the first is I've not quite got the URL rewrite/forwarding configured properly, and the second is airflow being unable to generate the 404 which sounds like a codebase problem
Ah I just got there! Had to modify the airflow base url to:
Copy code
- name: "AIRFLOW__WEBSERVER__BASE_URL"
          value: http://{{ .Values.hostname }}/airflow
So my final config for anyone interested: Nginx:
Copy code
location /airflow/ {	
                proxy_pass <http://airflow-ui:8080/airflow/>;	
                proxy_set_header Host $host;	
                proxy_redirect off;	
                proxy_http_version 1.1;	
                proxy_set_header Upgrade $http_upgrade;	
                proxy_set_header Connection "upgrade";	
            }

            location / {
                proxy_pass <http://meltano-ui:8000>;
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;
                proxy_set_header X-Forwarded-Host $server_name;
            }
K8 services:
Copy code
---
apiVersion: v1
kind: Service
metadata:
  name: airflow-ui
spec:
  selector:
    app: airflow-ui
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: meltano-ui
spec:
  selector:
    app: meltano-ui
  ports:
    - protocol: TCP
      port: 8000
      targetPort: 8000
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
Airflow container env vars:
Copy code
- name: "MELTANO_DATABASE_URI"
          value: {{ .Values.database_url }}/meltano
        - name: "AIRFLOW__CORE__SQL_ALCHEMY_CONN"
          value: {{ .Values.database_url }}/airflow
        - name: "MELTANO_READONLY"
          value: "1"
        - name: "AIRFLOW__WEBSERVER__BASE_URL"
          value: http://{{ .Values.hostname }}/airflow
        - name: "AIRFLOW__WEBSERVER__ENABLE_PROXY_FIX"
          value: "true"
a
Awesome! Glad you got it working 👍 This would make a good addition to the Meltano deployment docs
d
@niall_woodward I'm glad you got it working! I agree that this should be in the documentation 🙂 We don't have a dedicated Meltano UI page yet (see https://gitlab.com/meltano/meltano/-/issues/2325) but we can start with a few hints under https://meltano.com/docs/production.html#airflow-orchestrator or https://meltano.com/docs/production.html#meltano-ui to save the next person the trouble of finding out which options matter and which don't. Would you be up for contributing some notes to that page?
7 Views
Open in Slack
PreviousNext
Powered by