Hi, fellows! I am new to Meltano, sorry for the noob question. I am trying to set it up and this is the error that I receive on "meltano init PROJECT": ```[2020-10-04 155523,818] [23524|MainThread|root] [CRITICAL] Could not update meltano.yml: 'WindowsPath' object has no attribute 'decode' Traceback (most recent call last): File "c:\users\ivano\appdata\local\programs\python\python37\lib\runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "c:\users\ivano\appdata\local\programs\python\python37\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\Users\ivano\AppData\Local\Programs\Python\Python37\Scripts\meltano.exe\__main__.py", line 7, in <module> File "c:\users\ivano\appdata\local\programs\python\python37\lib\site-packages\meltano\cli\__init__.py", line 38, in main cli(obj={"project": None}) File "c:\users\ivano\appdata\local\programs\python\python37\lib\site-packages\click\core.py", line 764, in call return self.main(*args, **kwargs) File "c:\users\ivano\appdata\local\programs\python\python37\lib\site-packages\click\core.py", line 717, in main rv = self.invoke(ctx) File "c:\users\ivano\appdata\local\programs\python\python37\lib\site-packages\click\core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "c:\users\ivano\appdata\local\programs\python\python37\lib\site-packages\click\core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "c:\users\ivano\appdata\local\programs\python\python37\lib\site-packages\click\core.py", line 555, in invoke return callback(*args, **kwargs) File "c:\users\ivano\appdata\local\programs\python\python37\lib\site-packages\click\decorators.py", line 17, in new_func return f(get_current_context(), *args, **kwargs) File "c:\users\ivano\appdata\local\programs\python\python37\lib\site-packages\meltano\cli\params.py", line 23, in decorate return func(*args, **kwargs) File "c:\users\ivano\appdata\local\programs\python\python37\lib\site-packages\meltano\cli\initialize.py", line 46, in init project = init_service.init() File "c:\users\ivano\appdata\local\programs\python\python37\lib\site-packages\meltano\core\project_init_service.py", line 39, in init self.set_send_anonymous_usage_stats() File "c:\users\ivano\appdata\local\programs\python\python37\lib\site-packages\meltano\core\project_init_service.py", line 61, in set_send_anonymous_usage_stats store=SettingValueStore.MELTANO_YML, File "c:\users\ivano\appdata\local\programs\python\python37\lib\site-packages\meltano\core\settings_service.py", line 286, in set value, _ = self.set_with_metadata(*args, **kwargs) File "c:\users\ivano\appdata\local\programs\python\python37\lib\site-packages\meltano\core\settings_service.py", line 279, in set_with_metadata set_metadata = manager.set(name, path, value, setting_def=setting_def) File "c:\users\ivano\appdata\local\programs\python\python37\lib\site-packages\meltano\core\settings_store.py", line 301, in set self.log(f"Set path '{path}' in `meltano.yml`: {value!r}") File "c:\users\ivano\appdata\local\programs\python\python37\lib\contextlib.py", line 119, in exit next(self.gen) File "c:\users\ivano\appdata\local\programs\python\python37\lib\site-packages\meltano\core\settings_store.py", line 344, in update_config self.settings_service._update_meltano_yml_config(config) File "c:\users\ivano\appdata\local\programs\python\python37\lib\site-packages\meltano\core\project_settings_service.py", line 55, in _update_meltano_yml_config self.config_service.update_config(config) File "c:\users\ivano\appdata\local\programs\python\python37\lib\site-packages\meltano\core\config_service.py", line 46, in update_config meltano.extras = config File "c:\users\ivano\appdata\local\programs\python\python37\lib\contextlib.py", line 119, in exit next(self.gen) File "c:\users\ivano\appdata\local\progra…

@zlatan Welcome Zlatan! Per https://meltano.com/docs/installation.html#unix-like-environment, Windows is not currently supported officially, and WSL or Docker are recommended. Is using either of those an option for you? Of course we'd still like to support Windows natively if possible. Can you please create a new issue for this bug on https://gitlab.com/meltano/meltano/-/issues? From a quick look at the source code, I think we could resolve this by updating the

atomicwrites

package to version 1.4.0, which includes this bugfix: https://github.com/untitaker/python-atomicwrites/commit/2a91f2ef3e85b6a6c4dca3209a02eaba9b003059

Thanks for the response, Douwe! I've tried running Meltano through Docker, but then I issues with installing the tap-facebook. The error is that pip is lower version and I couldn't understand how can I upgrade meltano to use higher pip version in the Docker container.

@zlatan Is the pip issue similar to what's being discussed and addressed in https://gitlab.com/meltano/meltano/-/issues/2372?

It seem so. Is there a solution for it?

@zlatan A solution is being implemented in https://gitlab.com/meltano/meltano/-/merge_requests/1894 🙂 Can you share more about the tap-facebook issue you ran into? There may be a temporary workaround

Yup, give me some minutes to replicate the issue, and I will write back 🙂

You're missing the

--interactive

option on

docker run

🙂 See https://meltano.slack.com/archives/C013EKWA2Q1/p1601914844054500?thread_ts=1601912805.054200&amp;cid=C013EKWA2Q1

Worked, thanks.

@zlatan Ah, right. See https://meltano.slack.com/archives/CFG3C3C66/p1601914709053300?thread_ts=1601912826.051000&amp;cid=CFG3C3C66 🙂

@douwe_maan just followed the advice to change the pip_url for tap-facebook in meltano.yml to aiohttp=2.0.0 tap-facebook, but I receive this response:

C:\devABI\fb_project\try1>docker run --interactive -v %cd%:/projects -w /projects meltano/meltano install extractor tap-facebook
Installing 1 plugins...
Installing extractor 'tap-facebook'...
Extractor 'tap-facebook' could not be installed: failed to install plugin 'tap-facebook'.
Invalid requirement: 'aiohttp=2.0.0'
= is not a valid operator. Did you mean == ?
You are using pip version 18.1, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Failed to install plugin(s)

@zlatan My bad, you need 2

==

s there!

Ah, sorry for my illiteracy!

No worries, I fixed the comment 🙂

@douwe_maan Moving forward, I am getting excited! Installed and set up FB, it seems that it returns results now. I installed target-bigquery and meltano config target-bigquery list returns this:

C:\devABI\fb_project\try1>docker run --interactive -v %cd%:/projects -w /projects meltano/meltano config target-bigquery list
Traceback (most recent call last):
  File "/usr/local/bin/meltano", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/meltano/cli/__init__.py", line 38, in main
    cli(obj={"project": None})
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/meltano/cli/config.py", line 200, in list_settings
    full_config = settings.config_with_metadata(session=session, extras=load_extras)
  File "/usr/local/lib/python3.6/site-packages/meltano/core/settings_service.py", line 115, in config_with_metadata
    for setting_def in self.definitions(extras=extras):
  File "/usr/local/lib/python3.6/site-packages/meltano/core/settings_service.py", line 329, in definitions
    self._definitions, self.flat_meltano_yml_config
  File "/usr/local/lib/python3.6/site-packages/meltano/core/settings_service.py", line 87, in flat_meltano_yml_config
    return flatten(self._meltano_yml_config, "dot")
  File "/usr/local/lib/python3.6/site-packages/meltano/core/plugin/settings_service.py", line 135, in _meltano_yml_config
    **{f"_{k}": v for k, v in project_plugin.current_extras.items()},
TypeError: 'str' object is not a mapping

@zlatan Can you share the

meltano.yml

definition for

target-bigquery

?

Now it returns

C:\devABI\fb_project\try1>docker run --interactive -v %cd%:/projects -w /projects meltano/meltano config target-bigquery list
Traceback (most recent call last):
  File "/usr/local/bin/meltano", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/meltano/cli/__init__.py", line 38, in main
    cli(obj={"project": None})
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/meltano/cli/config.py", line 200, in list_settings
    full_config = settings.config_with_metadata(session=session, extras=load_extras)
  File "/usr/local/lib/python3.6/site-packages/meltano/core/settings_service.py", line 124, in config_with_metadata
    **kwargs,
  File "/usr/local/lib/python3.6/site-packages/meltano/core/settings_service.py", line 187, in get_with_metadata
    if setting_def and setting_def.is_extra:
  File "/usr/local/lib/python3.6/site-packages/meltano/core/setting_definition.py", line 126, in is_extra
    return self.name.startswith("_")
AttributeError: 'NoneType' object has no attribute 'startswith'

Ah, that doesn't look quite right; it looks like your newlines were picked up as

\r

😬 Can you change

namespace

to

bigquery

,

executable

to

target-bigquery

and remove

settings

entirely?

OK, will let you know of results

Note that you'll still need to configure

target-bigquery

, of course

yes, i'll try to get the command "meltano config target-bigquery list"

That won't list anything though, since Meltano doesn't know about the target's settings yet 🙂 Did you enter the setting names when

meltano add --custom

asked you about them?

Okay, I'll try now

Settings are these: https://github.com/RealSelf/target-bigquery/blob/master/config.sample.json

Oh, and where do the OAuth settings go?

Judging by the README, in a file named

client_secrets.json

in your project dir: https://github.com/RealSelf/target-bigquery#step-1-activate-the-google-bigquery-api

Ah, I'll get to reading. Thanks, man

@zlatan That doesn't look specific to Meltano, no. If you run

meltano --log-level=debug elt ...

, you should see the exact command arguments and env vars

target-bigquery

was invoked with. Is

GOOGLE_APPLICATION_CREDENTIALS

listed there?

Not found

How are you setting the

GOOGLE_APPLICATION_CREDENTIALS

?

Just tried it, but it still gives the same error. Tried to have "path" and without the ".

@zlatan That looks correct. It should work without the

TARGET_BIGQUERY_

prefix since I think the Google/BigQuery API package itself reads the env var. Did you say that the env vars do not show up in the

Env:

hash when you run

meltano --log-level=debug elt ...

? That's what we'll want to fix first. Can you try wrapping the values in

.env

in double quotes to make sure it's not a syntax issue?

Hi, Douwe! I'll try it out now

@zlatan If env vars defined in

.env

are not making it to the plugin when it's invoked, we're looking at a Meltano bug 😬 Can you please create an issue at https://gitlab.com/meltano/meltano/-/issues?

Yes, sure, I'll add it there

In the mean time, try this workaround: In your

meltano.yml

, under

settings

, add a dummy setting with `env: GOOGLE_APPLICATION_CREDENTIALS`:

settings:
- name: credentials_path
  env: GOOGLE_APPLICATION_CREDENTIALS

Then you can add

credentials_path: path-to-credentials.json

under

config:

, or keep setting

GOOGLE_APPLICATION_CREDENTIALS

through

.env

as you are doing now

OK, will try it now

Great, let me know how it goes 🙂 If you file the issue, I can make sure the workaround won't be needed in the future

Should look like this, correct?

Meltano is running inside Docker right? Then

C:/

wouldn't be available from inside the container 🙂

@douwe_maan you are a genius, my friend!

@zlatan That's great to hear! And thanks for the kind words 😄

Yup! I will upload it tomorrow first thing (it's 0:19 AM here, going to bed now). 🙂