Where do I find the secret sauce that maps environment variables to configuration parameters? I'm having some issues with

tap-postgres

currently that doesn't seem to find my

TAP_POSTGRES_SQLALCHEMY_URL

variable and barfs a KeyError because it had expected

user

to be defined. Thanks in advance for any assistance!

they are called settings. https://docs.meltano.com/reference/settings/#plugin-settings Secret sauce I think you're asking for is here, picture shows the easiest way to get to it from the hub https://github.com/meltano/hub/blob/main/_data/meltano/extractors/tap-postgres/meltanolabs.yml Could you share your meltano.yml ?

Yeah, I got a little further actually. I think my environment-specific configurations might have "occluded" the stuff in the

plugins:

section of the yaml...

you shouldnt' have to do that, but it works, if you'd share your meltano.yml we can point you better

Not clear how it's even working with

namespace*:* upstream

If I remove line 17 I get this error:

$ docker run -it --env-file meltano/.env-prod im-meltano -- invoke tap-postgres
2024-08-09T18:50:03.670163Z [info     ] Environment 'prod' is active
/project/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/paramiko/pkey.py:100: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be removed from this module in 48.0.0.
  "cipher": algorithms.TripleDES,
/project/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/paramiko/transport.py:259: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be removed from this module in 48.0.0.
  "class": algorithms.TripleDES,
Traceback (most recent call last):
  File "/project/.meltano/extractors/tap-postgres/venv/bin/tap-postgres", line 8, in <module>
    sys.exit(TapPostgres.cli())
  File "/project/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/project/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/project/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/plugin_base.py", line 80, in invoke
    return super().invoke(ctx)
  File "/project/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/project/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/project/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 500, in invoke
    tap = cls(
  File "/project/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/tap_postgres/tap.py", line 52, in __init__
    super().__init__(*args, **kwargs)
  File "/project/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 649, in __init__
    super().__init__(*args, **kwargs)
  File "/project/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 109, in __init__
    self.setup_mapper()
  File "/project/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 179, in setup_mapper
    self.mapper.register_raw_streams_from_catalog(self.catalog)
  File "/project/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/tap_postgres/tap.py", line 542, in catalog
    for stream in super().catalog.streams:
  File "/project/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 172, in catalog
    self._catalog = self.input_catalog or self._singer_catalog
  File "/project/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 325, in _singer_catalog
    for stream in self.streams.values()
  File "/project/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 134, in streams
    for stream in self.load_streams():
  File "/project/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 358, in load_streams
    for stream in self.discover_streams():
  File "/project/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/tap_postgres/tap.py", line 600, in discover_streams
    for catalog_entry in self.catalog_dict["streams"]:
  File "/project/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/tap_postgres/tap.py", line 526, in catalog_dict
    result["streams"].extend(self.connector.discover_catalog_entries())
  File "/usr/local/lib/python3.9/functools.py", line 993, in __get__
    val = self.func(instance)
  File "/project/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/tap_postgres/tap.py", line 422, in connector
    url = make_url(self.get_sqlalchemy_url(config=self.config))
  File "/project/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/tap_postgres/tap.py", line 337, in get_sqlalchemy_url
    username=config["user"],
KeyError: 'user'

but it's a python namespace, nothing to do with your schema

Oh ok.

target-postgres

uses the namespace variable to set the target schema, though.

https://docs.meltano.com/concepts/plugins/#load_schema-extra is what you want to use

Thanks!

The other thing with user

It's weird... That list was just VERY short moments ago

yes

I see

the namespace in that "magic" hub portion (which is actually in your lock file now) uses the namespace to store the settings

Thank you Derek

You can use a different namespace, but you'd have to define the settings yourself in your own meltano.yml file

Oh.... Means I should probably re-lock that and commit

so you'd copy that thing I sent over

yeah? So how would I undo my little namespace gambit here?

good, then

meltano install --clean

ah yeah, or just docker build

we could just do tap-postgres but it doesn't take that long

myeah

we're missing some settings in that file whops

I do need to set up a tap dev environment eitherway.. Figured that

tap-typeform

is kinda crusty and not really functional earlier today

I get the feeling, yeah that's the way I'd do it. Welcome to meltano, glad to have you with us 😄

So I'm using that for the time being, and planning to do some improvements

seems like you know meltano pretty well though

I'm pretending mostly 😛

we all just hallucinate right, that's the word they use now for it

Who's in charge of promoting/demoting plugins? The current 🥇 here is defunct fwiw: https://hub.meltano.com/extractors/tap-typeform (did file an issue: https://github.com/albert-marrero/tap-typeform/issues/3)

can you put an issue in here https://github.com/meltano/hub/issues ?

Yeah, that's strange. However I found it really really easy to reproduce

That one is just you can't run with python > 3.9

Basically

meltano init

then ``meltano add extractor tap-typeform`

easy fix

Thanks a lot for your help, I really appreciate it!