It seems that environments are now required and that is why Meltano #troubleshooting

It seems that "environments" are now required, and...

fred_reimer

02/08/2023, 5:55 PM

It seems that "environments" are now required, and that is why our scheduled elt runs are failing. When trying locally I get:

Copy code

$ meltano run tap-hourly target-jsonl
A Meltano environment must be specified. Set the `default_environment` option in `meltano.yml`, or the `--environment` CLI option.

That is what the completely non-helpful log message show here are probably referring to:

Copy code

{subprocess.py:85} INFO - Output:
[2023-02-08, 06:17:27 UTC] {subprocess.py:92} INFO - {"event": "'false' is not a valid EnvVarMissingBehavior", "exc_info": ["<class 'ValueError'>", "ValueError(\"'false' is not a valid EnvVarMissingBehavior\")", "<traceback object at 0x7f3a89e8dac0>"], "timestamp": "2023-02-08T06:17:27.304770Z", "level": "error"}
[2023-02-08, 06:17:27 UTC] {subprocess.py:92} INFO - Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
[2023-02-08, 06:17:27 UTC] {subprocess.py:92} INFO - join our friendly Slack community.

I will look, but is there any concern of our bookmarks not working properly if we "upgrade" to use environments?

Will Da Silva (Arch)

02/08/2023, 6:07 PM

Hi @fred_reimer. That bug was fixed in https://github.com/meltano/meltano/pull/7232. The fix will be included with the next Meltano release. fyi @ken_payne

Will Da Silva (Arch)

02/08/2023, 6:08 PM

Are you setting the env var

$MELTANO_FF_STRICT_ENV_VAR_MODE

anywhere? That might be the cause of the problem here

fred_reimer

02/08/2023, 6:09 PM

Oh so it's a bug, and we don't need to do anything and convert environments? No, that's the first I've seen that env var. It is not set anywhere in our environment.

fred_reimer

02/08/2023, 6:19 PM

Where is the variable documented? What are the valid options for it?

Will Da Silva (Arch)

02/08/2023, 6:22 PM

That env var lets you configure the feature flag for strict env var mode, i.e. the setting

ff.strict_env_var_mode

. The valid values are

'true'

'false'

, or any other value that are "truthy" or "falsey". Unfortunately they weren't being cast to booleans correctly, which is what the linked PR fixed.

Will Da Silva (Arch)

02/08/2023, 6:22 PM

I only encountered this issue when that env var was being set. I'm not sure what might cause it to occur otherwise, but I'll see what I can find.

fred_reimer

02/08/2023, 6:22 PM

Copy code

$ MELTANO_FF_STRICT_ENV_VAR_MODE=False meltano run tap-logzio-volume-hourly target-jsonl
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

'False' is not a valid EnvVarMissingBehavior

$ MELTANO_FF_STRICT_ENV_VAR_MODE=false meltano run tap-logzio-volume-hourly target-jsonl
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

'false' is not a valid EnvVarMissingBehavior

Will Da Silva (Arch)

02/08/2023, 6:23 PM

It needs to be unset for the older (buggy) versions of Meltano to work properly

Will Da Silva (Arch)

02/08/2023, 6:23 PM

There is no value you can set it to that won't cause an error, because it fails to cast it to a boolean

fred_reimer

02/08/2023, 6:23 PM

When it is unset I get an error that the environment is not defined.

Will Da Silva (Arch)

02/08/2023, 6:25 PM

Yeah, so either the env var is being set by something you're unaware of (possibly within the process?), or there are other ways for this bug to occur that I'm not aware of.

fred_reimer

02/08/2023, 6:26 PM

Just to be sure, is this feature setting related to:

Copy code

A Meltano environment must be specified. Set the `default_environment` option in `meltano.yml`, or the `--environment` CLI option.

Or is that something completely different? I don't want to assume that a Meltano "environment" is the same as this strict env var setting.

Will Da Silva (Arch)

02/08/2023, 6:27 PM

That error is a red herring. It doesn't have much at all to do with the actual error you're encountering. Sorry about that.

Will Da Silva (Arch)

02/08/2023, 6:29 PM

Are you able to install a different version of Meltano? A quick-and-dirty fix might be to install the latest version of Meltano from our GitHub, in which this bug should be resolved.

Will Da Silva (Arch)

02/08/2023, 6:29 PM

I can help with the installation, if that is an option

fred_reimer

02/08/2023, 6:36 PM

Like I said, we don't set that. We: • pull from meltano:latest-python3.9. • setup .env for DB passwords and such (no setting here) • Clone the repo with the meltano.yml for this specific job • run meltano install • run meltano invoke airflow tasks run DAGID extract_load SCHEDULE --local --subdir DAGS_FOLDER/meltano.py (this is generated by airflow) If we have a docker image tag or hash we can use that instead of latest. Latest is what got us into this mess in the first place, we needed to move to gitlab.com from a self-hosted, and it rebuilt the image and "upgraded" meltano automatically with the CI. DB migrations, and a bunch of headaches followed.

fred_reimer

02/08/2023, 7:28 PM

So I put a sleep in the pod that runs the worker (DAG / elt run), and there are no env vars like that, and nothing in .env in the project dir. If I do a meltano run tap... target... I don't get the error about the envvar, but do get one about not having an environment. If I do the airflow command, it errors out with the envvar error. So something with airflow.

fred_reimer

02/08/2023, 7:34 PM

Oh, I do see this as far as the envvar:

Copy code

$ meltano config meltano list
default_environment [env: MELTANO_DEFAULT_ENVIRONMENT] current value: None (default)
...
ff.strict_env_var_mode [env: MELTANO_FF_STRICT_ENV_VAR_MODE] current value: False (default)

Where would that be set? I certainly didn't set it.

Will Da Silva (Arch)

02/08/2023, 7:35 PM

That's saying that

MELTANO_FF_STRICT_ENV_VAR_MODE

can be used to set that setting, but that the current setting is the default, rather than coming from the env var.

Will Da Silva (Arch)

02/08/2023, 7:36 PM

Meltano passes lots of env vars to subprocesses, including stuff like

MELTANO_FF_STRICT_ENV_VAR_MODE

. I'm wondering if somehow it's exporting

MELTANO_FF_STRICT_ENV_VAR_MODE=false

, and then consuming that later. I don't yet know how that could be happening, but it's plausible.

fred_reimer

02/08/2023, 7:39 PM

Yea, I just need to get this working. I don't care how. It's prod, and we haven't run any hourly or daily jobs since yesterday. At some point we will have to do a database restore and pin the images to what was "latest" 10 months ago. Everything was working then, which was the last time we pushed the image.

Will Da Silva (Arch)

02/08/2023, 8:42 PM

@fred_reimer I've published Meltano v2.15.2 which should fix this issue for you. Could you please try it. https://pypi.org/project/meltano/2.15.2/ https://hub.docker.com/r/meltano/meltano/tags

fred_reimer

02/08/2023, 8:45 PM

Thank you. We are doing a restore now, as there are multiple issues, but will check this out if/when we intentionally upgrade in the future!

fred_reimer

02/08/2023, 11:37 PM

Restore complete, first jobs completed successfully, rest are playing catch-up.

Open in Slack

Previous Next