Hey Everyone I ve got a containerized deployment of Meltano Meltano #troubleshooting

Hey Everyone, I've got a containerized deployment ...

philip_johnson

09/04/2022, 4:09 PM

Hey Everyone, I've got a containerized deployment of Meltano / Airflow Orchestrator and running into something strange. Each time a job runs, the following unexpected things happen: • Airflow, can't seem to find the logs generated by Meltano - I'm not sure if additional configuration is needed to change where Airflow expects the logs to be since meltano stores them using a different naming convention?

Copy code

Log file does not exist: /project/.meltano/run/airflow/logs/dag_id=meltano_leaflink-to-postgres/run_id=manual__2022-09-04T15:46:20.480973+00:00/task_id=extract_load/attempt=1.log
*** Fetching from: <http://0001306f9ad1:8793/log/dag_id=meltano_leaflink-to-postgres/run_id=manual__2022-09-04T15:46:20.480973+00:00/task_id=extract_load/attempt=1.log>
*** !!!! Please make sure that all your Airflow components (e.g. schedulers, webservers and workers) have the same 'secret_key' configured in 'webserver' section and time is synchronized on all your machines (for example with ntpd) !!!!!
****** See more at <https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#secret-key>
****** Failed to fetch log file from worker. Client error '403 FORBIDDEN' for url '<http://0001306f9ad1:8793/log/dag_id=meltano_leaflink-to-postgres/run_id=manual__2022-09-04T15:46:20.480973+00:00/task_id=extract_load/attempt=1.log>'
For more information check: <https://httpstatuses.com/403>

• The tasks run for a few minutes then seem to automatically fail. Based on the logs in the Meltano UI, it looks as if they are just killed by something external - no errors • I'm able to run the pipeline without if I ssh in, run bash in the meltano-ui container and do something like

meltano run tap-leaflink target-postrgres

Any thoughts / advice?

philip_johnson

09/04/2022, 5:34 PM

Found an old thread and did the following to fix my issue: 1. Created my own dag that I can schedule outside of the meltano UI 2. Found the

base_log_folder

in the airflow config that was generated which happened to be

/project/.meltano/run/airflow/logs

3. Created a new volume and then shared it across the webserver and scheduler like so...

Copy code

...

services:
  
  # Non relevant config
  # ...

  airflow-scheduler:
    <<: *meltano-image
    command: invoke airflow scheduler
    environment:
      <<: *meltano-env
      <<: *airflow-env
    volumes:
      - meltano_elt_logs_data:/project/.meltano/logs/elt
      - airflow_logs_data:/project/.meltano/run/airflow/logs
    expose:
      - 8793
    depends_on:
      - meltano-system-db
      - airflow-metadata-db
    networks:
      - meltano
      - airflow
    restart: unless-stopped
  
  airflow-webserver:
    <<: *meltano-image
    command: invoke airflow webserver
    environment:
      <<: *meltano-env
      <<: *airflow-env
    volumes:
      - meltano_elt_logs_data:/project/.meltano/logs/elt
      - airflow_logs_data:/project/.meltano/run/airflow/logs

    expose:
      - 8080
    ports:
      - 8080:8080
    depends_on:
      - meltano-system-db
      - airflow-metadata-db
    networks:
      - meltano
      - airflow
    restart: unless-stopped
  
  # Non relevant config
  # ...

volumes:
  meltano_postgresql_data:
    driver: local
  meltano_elt_logs_data:
    driver: local
  airflow_postgresql_data:
    driver: local
  airflow_logs_data:
    driver: local

Open in Slack

Previous Next