https://linen.dev logo
Join Slack
Powered by
Hello, I am running a ELT postgres to jsonl. Every...
# troubleshooting
o
Hello, I am running a ELT postgres to jsonl. Everything looks fine but when I run it, I got this error:
Copy code
[info     ] Incremental state has been updated at 2022-02-02 21:59:49.354952.
2022-02-02T21:59:49.356159Z [error    ] Extraction failed              code=1 job_id=postgres-to-jsonl message=AttributeError: 'NoneType' object has no attribute 'get' name=meltano run_id=47a44ca5-792d-411a-a399-8127188e4919
2022-02-02T21:59:49.356876Z [info     ] ELT could not be completed: Extractor failed cmd_type=elt job_id=postgres-to-jsonl name=meltano run_id=47a44ca5-792d-411a-a399-8127188e4919 stdio=stderr
a
This could be a configuration error - hard to say with this error message. Did anything helpful come from context in the surrounding log file and/or via adding 
--log-level=debug
in the CLI?
If there's an error in your tap configuration, this might be able to be surfaced with something like 
meltano config tap-postgres test
o
Thanks for jumping in @aaronsteers The conexion is fine since I can see the tables with this command
Copy code
meltano select tap-postgres --list --all
I am running the test...
this is the return from the test
Copy code
oscar@Oscar-Gonzalez:~/meltano-projects/demo-project$ meltano config tap-postgres test
Plugin configuration is invalid
AttributeError: 'NoneType' object has no attribute 'get'
a
Thanks, @oscar_gonzalez! Can you try also with this slight alteration? 
meltano --log-level=debug config tap-postgres test
I'm hoping for an error message that tells us where to look.
o
sure thing...
Is not telling me more, but here it is
Copy code
2022-02-02T23:31:26.644113Z [debug    ] Plugin configuration is invalid
AttributeError: 'NoneType' object has no attribute 'get' 
Traceback (most recent call last):
  File "/home/oscar/.pyenv/versions/3.8.11/lib/python3.8/site-packages/meltano/cli/__init__.py", line 47, in main
    cli(obj={"project": None})
  File "/home/oscar/.pyenv/versions/3.8.11/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/oscar/.pyenv/versions/3.8.11/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/oscar/.pyenv/versions/3.8.11/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/oscar/.pyenv/versions/3.8.11/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/oscar/.pyenv/versions/3.8.11/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/oscar/.pyenv/versions/3.8.11/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/oscar/.pyenv/versions/3.8.11/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/oscar/.pyenv/versions/3.8.11/lib/python3.8/site-packages/meltano/cli/config.py", line 290, in test
    raise CliError("\n".join(("Plugin configuration is invalid", detail)))
meltano.cli.utils.CliError: Plugin configuration is invalid
AttributeError: 'NoneType' object has no attribute 'get'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/oscar/.pyenv/versions/3.8.11/lib/python3.8/site-packages/meltano/cli/__init__.py", line 55, in main
    raise CliError(str(err)) from err
meltano.cli.utils.CliError: Plugin configuration is invalid
AttributeError: 'NoneType' object has no attribute 'get'
Plugin configuration is invalid
AttributeError: 'NoneType' object has no attribute 'get'
maybe the tests?
a
I see 'Plugin configuration is invalid' a few times there. After scanning for and scrubbing any sensitive info (if applicable), could you post your 
meltano.yml
file or at least a snippet from it that covers the tap-postgres portion?
We can also open an issue at this point if you felt like dropping the logs + snippets into a dedicated gitlab ticket. Even if this is a configuration problem (which is what I suspect), we probably should try to improve the logging/messaging on our side so its easier to debug.
o
Copy code
send_anonymous_usage_stats: true
project_id: 2713e6be-e0f5-4594-8f3c-3269c107525a
plugins:
  extractors:
  - name: tap-postgres
    variant: transferwise
    pip_url: pipelinewise-tap-postgres
    config:
      dbname: replication
      host: host
      user: user
      port: 5432
      default_replication_method: INCREMENTAL
      password: password
      filter_schemas: replication
Thats the yml part...I removed the host/user/password
a
I don't see anything that looks incorrect from that snippet. Do you have other taps configured? If so, could you try the same test with another tap? (This will tell us if the error is being triggered based on something elsewhere in the file.)
It might be worthwhile to open a ticket with the full 
meltano.yml
file contents. (You can flag the issue confidential in addition to scrubbing sensitive info if that helps.)
o
I will test with another DB we have. The only thing I can think of is that this DB is a read only DB.
a
Read only should be fine. This error message (None has no attribute 'get') normally occurs when a blank or null value is passed in config, where actually a nested value is expected.
Another engineer took a quick look at the log and it looks like the error is coming from within the tap. It might be worth calling the tap directly with a config.json file as input. Is this the pipelinewise variant? If it turns out to be a tap bug (i.e. the same thing happening when invoked directly), you could log an issue in their repo: https://github.com/transferwise/pipelinewise-tap-postgres/issues
Sorry this one seems very tricky to debug 😬
Open in Slack
PreviousNext
Powered by