Has anyone else ran into issues with setting schedule specific env vars? I have meltano.yml setting like this

schedules:
- name: minute-hubspot-load
  interval: '* * * * *'
  job: tap-hubspot-to-target-redshift
  env:
    TARGET_REDSHIFT_DEFAULT_TARGET_SCHEMA: hubspot_staging_testv2

And it doesn't seem to take that env var. I do see, when I run

airflow scheduler

this output.

Considering task 'tap-hubspot target-redshift' of schedule 'minute-hubspot-load': {'name': 'minute-hubspot-load', 'interval': '* * * * *', 'cron_interval': '* * * * *', 'env': {'TARGET_REDSHIFT_DEFAULT_TARGET_SCHEMA': 'hubspot_staging_testv2'}, 'job': {'name': 'tap-hubspot-to-target-redshift', 'tasks': ['tap-hubspot target-redshift']}}

So I think it is being passed to the scheduler run, but it does not work. If I set that exact same env var in

.env

file, it works. So I think it's an issue with scheduler env var not working properly. Does anyone know how to resolve this issue with airflow? Thank you.

Hi @Kyunghwan Choi! How are you running the schedule?

Hello. I use

meltano invoke airflow scheduler

Gotcha. So you generated the DAG and have a

meltano.py

DAG file in your project, right? (I'm looking at https://github.com/meltano/airflow-ext/blob/main/files_airflow_ext/orchestrate/meltano.py to see which Meltano command ends up running there).

I just ran

meltano invoke airflow:initialize

which I believe is generating DAG.

Ok, I created https://github.com/meltano/meltano/issues/8730 to track this and try to reproduce it.

hmm.. not sure, but when I delete all the settings and set it in .env file, it works. If it's only defined under scheduler, it uses the 'tap-hubspot' as a schema name, so it's like not setting it anywhere. So I don't think I'm defining it anywhere else, because it should have used some other value as a schema name, if there was another setting that was being utilized.

sure, it's

version: 1
default_environment: dev
project_id: 065f56c4-8058-4518-a620-7d628526532c
environments:
- name: dev
- name: staging
- name: prod
send_anonymous_usage_stats: false
plugins:
  extractors:
  - name: tap-pypistats
    namespace: tap_pypistats
    variant: edgarrmondragon
    pip_url: tap-pypistats==0.0.3
    settings_group_validation:
    - - packages
    settings:
    - kind: array
      label: Packages
      name: packages
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl
venv:
  backend: uv
jobs:
- name: pypistats-to-jsonl
  tasks:
  - tap-pypistats target-jsonl
schedules:
- name: weekly-pypistats
  interval: '@weekly'
  job: pypistats-to-jsonl
  env:
    TAP_PYPISTATS_PACKAGES: '["uv"]'

hmm.. yes, odd. Looks similar. Let me see if I just need to put the values in quotes, though I highly doubt it..