https://linen.dev logo
Join Slack
Powered by
I’m trying to get a schedule and Airflow running f...
# troubleshooting
b
I’m trying to get a schedule and Airflow running for a Postgres to S3 job and am getting the following error.
Copy code
icegateway_2_s3 [FOUND-13212-ksedw-export] $ meltano invoke airflow scheduler -D

2022-06-17T19:12:54.742525Z [info     ] Environment 'dev' is active
2022-06-17T19:12:59.889509Z [error    ] Traceback (most recent call last):
  File "/Users/bhughes/Documents/repos/data.architecture/meltano/icegateway_2_s3/.meltano/orchestrators/airflow/venv/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/Users/bhughes/Documents/repos/data.architecture/meltano/icegateway_2_s3/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow
/__main__.py", line 40, in main
    args.func(args)
  File "/Users/bhughes/Documents/repos/data.architecture/meltano/icegateway_2_s3/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow
/cli/cli_parser.py", line 47, in command
    func = import_string(import_path)
  File "/Users/bhughes/Documents/repos/data.architecture/meltano/icegateway_2_s3/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow

/utils/module_loading.py", line 32, in import_string
    module = import_module(module_path)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/importlib/__init__.py", line 127, in import_
module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/bhughes/Documents/repos/data.architecture/meltano/icegateway_2_s3/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow
/cli/commands/db_command.py", line 24, in <module>
    from airflow.utils import cli as cli_utils, db
  File "/Users/bhughes/Documents/repos/data.architecture/meltano/icegateway_2_s3/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow
/utils/db.py", line 26, in <module>
    from airflow.jobs.base_job import BaseJob  # noqa: F401
  File "/Users/bhughes/Documents/repos/data.architecture/meltano/icegateway_2_s3/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow
/jobs/__init__.py", line 22, in <module>
    import airflow.jobs.scheduler_job  # noqa
  File "/Users/bhughes/Documents/repos/data.architecture/meltano/icegateway_2_s3/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow
/jobs/scheduler_job.py", line 34, in <module>
    from setproctitle import setproctitle
ImportError: dlopen(/Users/bhughes/Documents/repos/data.architecture/meltano/icegateway_2_s3/.meltano/orchestrators/airflow/venv/lib/python3.8/site-pack
ages/setproctitle.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '_Py_GetArgcArgv'

Airflow metadata database could not be initialized: `airflow initdb` failed
Correction: I’m running through docker, so this is the command/error message: icegateway_2_s3 [FOUND-13212-ksedw-export] $ docker run -v $(pwd):/project -w /project meltano/meltano add orchestrator airflow 2022-06-17T192549.145606Z [info ] Environment ‘dev’ is active Plugin definition is already locked at /project/plugins/orchestrators/airflow--apache.lock. You can remove the file manually to avoid using a stale definition. Installing orchestrator ‘airflow’... [Errno 2] No such file or directory: ‘/project/.meltano/orchestrators/airflow/venv/bin/python’
a
Are you running this from the root dir of your meltano project?
b
Yes.
Copy code
icegateway_2_s3 [FOUND-13212-ksedw-export] $ pwd
/Users/bhughes/Documents/repos/data.architecture/meltano/icegateway_2_s3
icegateway_2_s3 [FOUND-13212-ksedw-export] $ ls
README.md               extract                 meltano.yml             orchestrate             plugins                 transform
analyze                 load                    notebook                output                  requirements.txt
a
Hm so Q1 Why run add orchestrator airflow if you already have it?
Does running the same docker command with invoke airflow —version
Instead of add orchestrator work?
If that doesn't work, use the install command instead of the add command.
b
I was trying to follow the documentation to get Airflow (but primarily the schedule) up and running. I’ve done this 2-3 times with other projects, so maybe it’s already there? Can you tell me more about how the Airflow plugin works? Do I really need it, or can I run a simple job/schedule w/o Airflow?
a
It's kind of heavy so it depends on your deployment pattern. You could technically just cron job the meltano run command on a posix system.
b
icegateway_2_s3 [FOUND-13212-ksedw-export] $ docker run -v $(pwd):/project -w /project meltano/meltano invoke airflow --version 2022-06-17T202504.478155Z [info ] Environment ‘dev’ is active Executable ‘airflow’ could not be found. Orchestrator ‘airflow’ may not have been installed yet using 
meltano install orchestrator airflow
, or the exe cutable name may be incorrect.
a
Oh okay. Try meltano discover orchestrators
b
2022-06-17T202610.665478Z [info ] Environment ‘dev’ is active Orchestrators airflow
a
Yeah it seems you need to run install command.
b
ok
a
The way it works is you ADD a plug-in to your project which adds a lock file and the specification. It's still up to you to INSTALL it. The install procedure creates a virtual environment. Meltano abstracts executing sub processes from these virtual environments.
b
What is the context for the add/install? Is Airflow added/installed into the Docker container or on the host machine (i.e. my laptop)? Got this msg: icegateway_2_s3 [FOUND-13212-ksedw-export] $ docker run -v $(pwd):/project -w /project meltano/meltano install orchestrator airflow 2022-06-17T203036.669624Z [info ] Environment ‘dev’ is active Installing 1 plugins... Installing orchestrator ‘airflow’... [Errno 2] No such file or directory: ‘/project/.meltano/orchestrators/airflow/venv/bin/python’
a
Yes it installs in docker but because you have a volume bind mount, it should show up on your file system.
Hence the parity between the two. And the $pwd mount.
b
Yes. I see the /orchestarte/dags folder.
a
Cool so that's evidence the file bundle is shipping with the orchestrator and it's effecting changes on your local file system when ran from docker.
b
How about this error? [Errno 2] No such file or directory: ‘/project/.meltano/orchestrators/airflow/venv/bin/python’
a
Use projects plurally in the command
for the -v and for the -w and run the install command again
You should have a .meltano directory. The orchestrate dags is NOT the source code for airflow but just where it loads jobs.
b
Same error: cegateway_2_s3 [FOUND-13212-ksedw-export] $ docker run -v $(pwd):/projects -w /projects meltano/meltano install orchestrator airflow 2022-06-17T204713.232705Z [info ] Environment ‘dev’ is active Installing 1 plugins... Installing orchestrator ‘airflow’... [Errno 2] No such file or directory: ‘/projects/.meltano/orchestrators/airflow/venv/bin/python’
a
I'm a bit stumped. The long shot is to run 
docker exec -it -v … -w … meltano/meltano — /bin/bash
And do stuff in the container till we find out why there is no .meltano dir able to be created
b
Ok. I’ll try that or stick with cron for the time being.
Thanks for all your help.
3 Views
Open in Slack
PreviousNext
Powered by