https://linen.dev logo
Join Slack
Powered by
I also find it confusing when adding an extractor ...
# announcements
m
I also find it confusing when adding an extractor via a custom tap (so in my case tap-mysql) what the identifiers (e.g. name and namespace) are. I used the command 
meltano add --custom extractor tap-mysql
as the example on the homepage uses 
tap-***
as well so I assumed the argument in that command was the type of tap I was adding. This isn’t the case, the argument is the identifier of the integration (so e.g. multiple mysql integrations would be added as ``meltano add --custom extractor random-name-1, meltano add --custom extractor random-name-2`` )? then the first line of the wizard says 
The 'namespace' refers to the data source name.
As a result I thought this was the place where I was supposed to name the integration (random-name-1). Then the executable defaults to the namespace, while it would be better to default to the package name (e.g. tap-mysql). The fact that the ’namespace` is the target schema to be used is also a bit weird to me. I expect this to be part of the target configuration and not to be in the tap. It should be possible to choose where the data is being written at the side of the target, the tap isn’t concerned with that?
d
Thanks a lot for that feedback; this part of the documentation (or just the behavior of 
meltano add --custom
itself) could definitely be clearer as far as the difference between the plugin name and the namespace is concerned. I'll create an issue for this tomorrow (unless you beat me to it 😄)
As far as using the namespace for the schema, this made sense in the use cases we've been using Meltano in ourselves, when the L stage was always followed by a T stage that would essentially pull the different namespace schemas together into a new 
analytics
schema, but it's definitely counter to expectations (and undesirable) in a generic ELT tool. You can override the 
schema
value that is passed to 
target-postgres
by setting the 
MELTANO_LOAD_SCHEMA
environment variable, but I don't think that's even documented right now. Do you think it would be acceptable to keep the current default behavior if it's actually documented and easy for people to figure out how to override the schema?
a
I found this confusing as well. I think I could get used to it, but just changing the wording from "namespace" would help a lot
I expected to run 
meltano add --custom extractor <official-tap-name>
but as I learn more I see why that wouldn't be ideal
d
Open in Slack
PreviousNext
Powered by