How does meltano detect if a plugin is custom nor ...
# announcements
j
How does meltano detect if a plugin is custom nor not? I explicitly set
namespace:
on
tap-gitlab
(the meltano fork from discovery.yml) and now it tells me that the tap doesn't support state and is no longer doing incremental runs. My hunch is that now meltano is treating it as a custom plugin and wants me to set
capabilities:
d
@julian_knight You've got it: Meltano will only look for a plugin's definition in
discovery.yml
if the entry in
meltano.yml
does not have a
namespace
. So setting a
namespace
means it'll assume your local/custom definition is the complete one.
What are you trying to accomplish by changing the namespace? There may be a more direct way of accomplishing that 🙂
There are plans to untangle the namespace mess, by the way: https://gitlab.com/meltano/meltano/-/issues/2208#note_388529270
j
Ultimately I want to change the schema that gets used in
target-postgres
I think in the short term we're happy to treat all plugins as
--custom
- it allows us to standarize the implementation and switch to forks of taps easier
I suppose I can set the schema with
metadata
, right?
d
@julian_knight Have you considered directly setting
target-postgres
's
schema
setting, in
meltano.yml
or using
PG_SCHEMA
when running this pipeline?
The extractor's preferred schema can not currently be set using an extra, but there's an issue for that: https://gitlab.com/meltano/meltano/-/issues/2282 🙂
As for setting the schema with metadata, you could if the loader supports that!
j
Got it. I do like having different schemas per tap to avoid namespace conflicts. It seems like the easiest way short term is to use custom plugins. Thanks!
d
Sounds good!