With

meltano v1.42.0

, I am getting the below error while running Airflow.

[2020-08-01 17:40:50,372] {models.py:273} INFO - Filling up the DagBag from /Users/ranand/meltano-projects/demo_project/orchestrate/dags
Traceback (most recent call last):
  File ".meltano/run/bin", line 11, in <module>
    load_entry_point('meltano==1.42.0', 'console_scripts', 'meltano')()
  File "/Users/ranand/meltano-projects/.venv/lib/python3.7/site-packages/meltano/cli/__init__.py", line 38, in main
    cli(obj={"project": None})
  File "/Users/ranand/meltano-projects/.venv/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/ranand/meltano-projects/.venv/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/ranand/meltano-projects/.venv/lib/python3.7/site-packages/click/core.py", line 1134, in invoke
    Command.invoke(self, ctx)
  File "/Users/ranand/meltano-projects/.venv/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/ranand/meltano-projects/.venv/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/ranand/meltano-projects/.venv/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/ranand/meltano-projects/.venv/lib/python3.7/site-packages/meltano/cli/cli.py", line 31, in cli
    project = Project.find()
  File "/Users/ranand/meltano-projects/.venv/lib/python3.7/site-packages/fasteners/lock.py", line 293, in wrapper
    return f(self, *args, **kwargs)
  File "/Users/ranand/meltano-projects/.venv/lib/python3.7/site-packages/meltano/core/project.py", line 109, in find
    cls.activate(project)
  File "/Users/ranand/meltano-projects/.venv/lib/python3.7/site-packages/fasteners/lock.py", line 293, in wrapper
    return f(self, *args, **kwargs)
  File "/Users/ranand/meltano-projects/.venv/lib/python3.7/site-packages/meltano/core/project.py", line 77, in activate
    project.run_dir().joinpath("bin").symlink_to(executable)
  File "/Users/ranand/meltano-projects/.venv/lib/python3.7/site-packages/meltano/core/utils/__init__.py", line 233, in decorate
    path = func(*args, **kwargs)
  File "/Users/ranand/meltano-projects/.venv/lib/python3.7/site-packages/meltano/core/project.py", line 201, in run_dir
    return self.meltano_dir("run", *joinpaths)
  File "/Users/ranand/meltano-projects/.venv/lib/python3.7/site-packages/meltano/core/utils/__init__.py", line 236, in decorate
    _, ext = os.path.splitext(path)
  File "/usr/local/bin/../Cellar/python/3.7.7/bin/../Frameworks/Python.framework/Versions/3.7/lib/python3.7/posixpath.py", line 122, in splitext
    p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not PosixPath
[2020-08-01 17:40:51,654] {models.py:377} ERROR - Failed to import: /Users/ranand/meltano-projects/demo_project/orchestrate/dags/meltano.py

@rahul_anand I think something else you’ve got installed in your Meltano venv may be bringing in an old version of `pathlib`; see https://github.com/psf/black/issues/407#issuecomment-405325781

I will try to reproduce this. But if I remember correctly I have seen this every time I install airflow in meltano venv.

@rahul_anand Hmm, did you install Airflow right into the Meltano venv, or using

meltano add orchestrator airflow

?

I did

meltano add orchestrator airflow

@rahul_anand Do you see the same error when you run

.meltano/run/bin schedule list --format=json

directly from your project dir?

@douwe_maan This is what I see:

(.venv) (base) ranand@RANAND-PRO demo_project_bak % ls -dl /Users/ranand/meltano-projects/.venv/lib/python3.7/site-packages/pathlib*
drwxr-xr-x  7 ranand  staff    224 Jul 30 11:33 /Users/ranand/meltano-projects/.venv/lib/python3.7/site-packages/pathlib-1.0.1-py3.7.egg-info
-rw-r--r--  1 ranand  staff  41481 Sep  3  2014 /Users/ranand/meltano-projects/.venv/lib/python3.7/site-packages/pathlib.py
drwxr-xr-x  4 ranand  staff    128 Jun  7 21:18 /Users/ranand/meltano-projects/.venv/lib/python3.7/site-packages/pathlib2
drwxr-xr-x  9 ranand  staff    288 Jun  7 21:18 /Users/ranand/meltano-projects/.venv/lib/python3.7/site-packages/pathlib2-2.3.5.dist-info

Airflow and Meltano expect different versions of various requirements, and neither is necessarily compatible with the different version the other requires. If both are installed into the same virtual environment with the same shared set of requirements (of which only 1 version can exist at a time), who knows what'll happen 🙂 In this case, it looks like Airflow requires https://pypi.org/project/pathlib/ (in some cases?), while Meltano depends on the newer version in Python 3.4+'s stdlib.

Sure. Let me try this.

@rahul_anand Awesome! Happy I could help 🙂