https://linen.dev logo
Join Slack
Powered by
Question: Is `env:` block `PYTHONPATH` ignored for...
# troubleshooting
k
Question: Is 
env:
block 
PYTHONPATH
ignored for utility plugins? (i.e. Airflow plugin) I'm running into an issue where I need my Airflow utility plugin to be able to import other Python modules rather than its default visible path – airflow/plugin, airflow/dags and airflow/plugins folders. Thus, an extra 
PYTHONPATH
is needed to help Airflow recognizes the module. I tried adding a 
PYTHONPATH
to the utility's 
env:
block in 
meltano.yml
, but it seems to be ignored or overwritten when Meltano activates the utility's venv. 
sys.path
does not contain the paths I set. Here's my debug process: 1. Test for 
env:
block processing: I added a test variable (
MY_TEST_VAR: "It works!"
) to the 
env:
block and printed it -> 
MY_TEST_VAR
was printed, confirming the 
env:
block is being read. 2. Test `PYTHONPATH`: I set 
PYTHONPATH
in the same 
env:
block -> The paths from 
PYTHONPATH
did not appear in 
sys.path
. This suggests me that Meltano is specifically overwriting or ignoring the 
PYTHONPATH
variable.
Copy code
plugins:
  utilities:
    - name: tester
      pip_url: ...
      commands:
        check-pythonpath:
          executable: python
          args: -c "import sys, pprint, os; pprint.pprint(sys.path); print(f\"MY_TEST_VAR: {os.environ.get('MY_TEST_VAR')}\")"
      env:
        MY_TEST_VAR: "It works!" # This works
        PYTHONPATH: /path/to/my/project # This does not
results in
Copy code
2025-10-20T10:27:29.250809Z [info     ] Using logging configuration from logging.yaml
2025-10-20T10:27:29.256688Z [info     ] Environment 'dev' is active
['',
 '/Users/khoa.nguyen/.asdf/installs/python/3.12.8/lib/python312.zip',
 '/Users/khoa.nguyen/.asdf/installs/python/3.12.8/lib/python3.12',
 '/Users/khoa.nguyen/.asdf/installs/python/3.12.8/lib/python3.12/lib-dynload',
 '/Users/khoa.nguyen/work/dataops/.meltano/utilities/testers/venv/lib/python3.12/site-packages'
]
MY_TEST_VAR= It works!
🤔 It seems counter-intuitive for 
PYTHONPATH
to be handled differently than other env vars. Is this the expected behaviors?
✅ 1
v
I'm not certain on your question in regards to why it's different, but here's an easy solution for python version differences per plugin https://docs.meltano.com/reference/settings/#python Should help!
Guessing that's not quite what you're after either Really looks like you're trying to do something specific with commands but don't want to use meltano's venv setup?
e
Yeah, Meltano removes 
PYTHONPATH
from the plugin's run environment to avoid conflicts between Meltano's own and the plugin's virtualenv: https://github.com/meltano/meltano/blob/7c94ed9db04ba2616bc6ca5a924b18a37cf7516c/src/meltano/core/plugin_invoker.py#L424-L426. Removing 
PYTHONPATH
might not be necessary, but ideas (or PRs 🙂) are welcome!
k
Really looks like you're trying to do something specific with commands but don't want to use meltano's venv setup?
I wish to add another directory to 
PYTHONPATH
per instructed in Airflow documentation such that I can migrate my shared module under airflow/dags/utils to airflow/utils – for example Mozilla team appends their entire project to Airflow's PYTHONPATH.
Yeah, Meltano removes 
PYTHONPATH
from the plugin's run environment to avoid conflicts between Meltano's own and the plugin's virtualenv
😅 I see – might I ask if there are still workaround/alternative way to append 
PYTHONPATH
for a Meltano plugin (i.e. Airflow)?
e
Can you try adding this to your Airflow plugin:
Copy code
settings:
- env: PYTHONPATH
  kind: string
  label: Airflow PYTHONPATH
  name: pythonpath
config:
  pythonpath: /path/to/python/modules
👀 1
k
> Can you try adding this to your Airflow plugin: Unfortunately, it didn't work 😕 – same output as written above, MY_TEST_VAR is printed out but path isn't included as part of the array
Perhaps, the 
settings.pythonpath
is added to the dictionary on line 416 https://github.com/meltano/meltano/blob/7c94ed9db04ba2616bc6ca5a924b18a37cf7516c/src/meltano/core/plugin_invoker.py#L416, but it's popped in next few code of lines anyway haha
I have found a solution to append extra 
PYTHONPATH
– similarly to append Python path on image build time, we can append the path at Airflow's initialization which runs after Meltano invoke/run command but before Airflow actually executes anything.
Copy code
# orchestrate/airflow/config/airflow_local_settings.py

# Add AIRFLOW_HOME to Python path
sys.path.append(os.getenv("AIRFLOW_HOME"))
🙌 2
🫶 Thank you guys for support and seeding the idea!
❤️ 2
4 Views
Open in Slack
PreviousNext
Powered by