I m taking another crack at getting an Airflow schedule runn Meltano #troubleshooting

I’m taking another crack at getting an Airflow sch...

blake_hughes

07/20/2022, 9:11 PM

I’m taking another crack at getting an Airflow schedule running, but I must be missing something. Invoking airflow seems to work fine. Running the schedule using the meltano schedule run command works. It seems no matter what I set the interval to (have tried hourly and cron expressions), the schedule doesn’t work. The job is never kicked off. I’m using the meltano 2.0 docker container. Any help would be appreciated.

blake_hughes

07/20/2022, 9:13 PM

Fyi…

$ docker run -v $(pwd):/project -w /project meltano/meltano --environment=prod invoke airflow scheduler

2022-07-20T210044.731757Z [info ] Environment ‘prod’ is active ____________ _____________ __ |__( )_________ / /________ __ __ /| |_ /__ ___/_ /_ _ / _ \ | /| / / _ _ | / _ / _ __/ _ / / /_/ /_ |/ |/ / _/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/ * Tip: There are .env or .flaskenv files present. Do “pip install python-dotenv” to use them. * Serving Flask app “airflow.utils.serve_logs” (lazy loading) * Environment: production WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Debug mode: off [2022-07-20 210050,430] {scheduler_job.py:1266} INFO - Starting the scheduler [2022-07-20 210050,431] {scheduler_job.py:1271} INFO - Processing each file at most -1 times [2022-07-20 210050,435] {dag_processing.py:254} INFO - Launched DagFileProcessorManager with pid: 23 [2022-07-20 210050,437] {_internal.py:113} INFO - * Running on http://0.0.0.0:8793/ (Press CTRL+C to quit) [2022-07-20 210050,438] {scheduler_job.py:1835} INFO - Resetting orphaned tasks for active dag runs [2022-07-20 210050,455] {settings.py:51} INFO - Configured default timezone Timezone(‘UTC’) [2022-07-20 210050,464] {dag_processing.py:529} WARNING - Because we cannot use more than 1 thread (parsing_processes = 2 ) when using sqlite. So we set parallelism to 1. [2022-07-20 210550,478] {scheduler_job.py:1835} INFO - Resetting orphaned tasks for active dag runs [2022-07-20 211050,508] {scheduler_job.py:1835} INFO - Resetting orphaned tasks for active dag runs

jean_sahlberg

07/20/2022, 11:11 PM

are you running airflow webserver?

blake_hughes

07/21/2022, 1:52 PM

I haven’t been able to get webserver running. It starts but it won’t come up in the browser.

jean_sahlberg

07/21/2022, 5:30 PM

try this:

docker run -v $(pwd):/project -w /project -p 8080:8080 meltano/meltano --environment=prod invoke airflow webserver

blake_hughes

07/21/2022, 5:37 PM

That worked for bringing up the Airflow UI. Thanks. My DAG is not there.

jean_sahlberg

07/21/2022, 6:10 PM

are you using the “files-airflow” to generate the dag?

jean_sahlberg

07/21/2022, 6:11 PM

if you didn’t add that it won’t work

jean_sahlberg

07/21/2022, 6:11 PM

just run

meltano add files airflow

blake_hughes

07/21/2022, 6:36 PM

I’ve tried various flavors of that command, but I keep getting this error: icegateway_2_athena [FOUND-9975-add-job] $ docker run -v $(pwd):/project -w /project meltano/meltano add files airflow 2022-07-21T183341.985251Z [info ] Environment ‘dev’ is active File bundle ‘airflow’ is not known to Meltano

aaron_phethean

07/21/2022, 6:55 PM

Nearly! Try ‘meltano add files files-airflow’ https://hub.meltano.com/files/files-airflow

blake_hughes

07/21/2022, 8:53 PM

That did it. The schedule is now working. Thanks for the help.

2 Views

Open in Slack

Previous Next