https://linen.dev logo
Join Slack
Powered by
Hello, it's me again) Trying to run airflow job g...
# troubleshooting
a
Hello, it's me again) Trying to run airflow job generated by 'meltano job add' ... but task fails. Need help Details inside
meltano.yaml
Copy code
version: 1
default_environment: dev
project_id: 183d71ea-2fe8-44fe-93bc-1bc9d285e5d6
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-stackexchange
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-stackexchange.git>
    config:
      tags:
      - python
      start_date: 1674981000
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl
  - name: target-postgres
    variant: transferwise
    pip_url: pipelinewise-target-postgres
    config:
      host: localhost
      port: 5432
      user: meltano
      dbname: postgres
      add_metadata_columns: true
  orchestrators:
  - name: airflow
    variant: apache
    pip_url: apache-airflow==2.1.2 --constraint <https://raw.githubusercontent.com/apache/airflow/constraints-2.1.2/constraints-${MELTANO__PYTHON_VERSION}.txt>
  files:
  - name: files-airflow
    variant: meltano
    pip_url: git+<https://github.com/meltano/files-airflow.git>
jobs:
- name: tap-stackexchange-to-target-postgres
  tasks:
  - - tap-stackexchange
    - target-postgres
schedules:
- name: minutely-stackexchange-load
  interval: '*/1 * * * *'
  job: tap-stackexchange-to-target-postgres
it generates this dag
generated_dag.txt
Task's logs
it's running command
Copy code
Running command: ['bash', '-c', 'cd /home/AlexRaz/meltano_projects/try2; .meltano/run/bin run tap-stackexchange target-postgres']
and gets
Copy code
'false' is not a valid EnvVarMissingBehavior
when I run this command via VM's console it's perfectly fine
Copy code
cd /home/AlexRaz/meltano_projects/try2; .meltano/run/bin run tap-stackexchange target-postgres
I can't find working examples for EL-flows, without dbt transformations
message has been deleted
w
@aleksei_razvodov could you please log this as a bug at https://github.com/meltano/meltano/issues EDIT: I have opened https://github.com/meltano/meltano/issues/7231 to track this issue
I'll try to reproduce this and fix it as soon as possible
The error seems to be that 
strict_env_var_mode = 'false'
, i.e. that it has a 
str
value when it ought to be a 
bool
. Now to figure out how that happened...
Looks like your 
strict_env_var_mode
is set by the env var 
MELTANO_FF_STRICT_ENV_VAR_MODE
I've opened https://github.com/meltano/meltano/issues/7231 to track this issue
a
Thanks, I appreciate it. @ me if I can provide additional info and help somehow
w
@aleksei_razvodov as a workaround, if you don't set the 
MELTANO_FF_STRICT_ENV_VAR_MODE
env var it shouldn't error. Do you know where the env var is being set?
a
@Will Da Silva (Arch) I'm not setting it explicitly. And 
echo $MELTANO_FF_STRICT_ENV_VAR_MODE
equals empty string, like any unset env var. Where I can check it?
a
@aleksei_razvodov - do you also have a 
.env
file by chance and can you double check there's not a value for this env var there?
a
@aaronsteers checked, only postgres password
any other ideas? airflow still doesn't work
a
@aleksei_razvodov - We've just released Meltano v2.15.2 which contains this fix. Can you confirm if rerunning with the latest version resolves your issue?
And secondly, we never discovered why 
MELTANO_FF_STRICT_ENV_VAR_MODE
was being loaded into your project. We're doing a post-mortem to try to identify possible root causes. Another user ran into this when using Airflow so we're wondering if the Airflow process is hiding some config perhaps. I know you already checked your Meltano .env and I double-checked your DAG files above. Do you mind checking your Airflow configuration to see if that environment variable is declared somewhere in the Airflow server or processes?
Open in Slack
PreviousNext
Powered by