https://linen.dev logo
Join Slack
Powered by
Is this command supposed to work if I am pulling i...
# singer-tap-development
b
Is this command supposed to work if I am pulling in my tap through a pip_url from github that hasnt been published to meltano hub yet?
Copy code
meltano select tap-canvas --list --all
the tap supports discovery and catalog features
r
Have you listed 
capabilities
in the definition?
b
Are you referring to on the meltano project side, like putting
Copy code
capabilities:
    - stream_maps
    - catalog
    - discover
into meltano.yml or is there something else I should be doing on the tap side. I looked through some examples and didn't see them declaring the capabilities in any particular file, currently I just have this:
Copy code
capabilities = [
        "catalog",
        "discover",
        "stream_maps", 
    ]
in my tap.py
r
Are you referring to on the meltano project side
Yep, that looks fine (
catalog
and 
discover
important in this case).
in my tap.py
You shouldn't need to override the defaults: https://github.com/meltano/sdk/blob/fddd5cfb0c0e419251167980e75f3976d1ef3577/singer_sdk/tap_base.py#L69-L79
What are you seeing from
Copy code
meltano select tap-canvas --list --all
?
Can you share you 
discover_streams
implementation also?
b
Sure this is what I get from meltano select tap-canvas --list --all
Copy code
CliError: Cannot list the selected attributes: Could not find catalog. Verify that the tap supports discovery mode and advertises the `discover`     
capability as well as either `catalog` or `properties`
and what I have in discover_streams: 
def discover_streams(_self_) -> List[Stream]:
"""Return a list of discovered streams."""
return [stream_class(_tap_=self) for stream_class in STREAM_TYPES]
r
Do you get output from
Copy code
meltano invoke --dump catalog <tap>
?
or
Copy code
meltano invoke <tap> --discover
?
b
For the first I get: CliError: Could not find catalog For the second I do get the catalog back
r
Can you share the entire plugin definition entry in your 
meltano.yml
?
b
Copy code
- name: tap-canvas
    namespace: tap_canvas
    pip_url: 
      git+<https://github.com/brunoarnabar/tap-canvas.git@main#egg=tap-canvas>
    capabilities:
    - state
    - catalog
    - discover
    select:
        - "!courses.*"
    config:
        record_limit: 10
this is what I've got
r
I tried that and it works fine. I have a feeling you just need to reinstall the plugin to get it working.
Copy code
meltano install --clean tap-canvas
More recent versions of Meltano detect when the 
pip_url
has changed and auto-reinstall. Which version are you using?
👍 1
I can see that you forked the original 
tap-canvas
, which is why I'm assuming the plugin has been installed previously and the 
pip_url
has changed.
b
yes is there a better way to test forked taps that have not been merged into meltano hub?
And I do see the select list showing, so I needed to explicitly name then in the yml for them to show? They don't get automatically pulled from the tap?
r
I think it's just change the 
pip_url
and reinstall. There is also 
meltano add --from-ref
which might be of interest: https://docs.meltano.com/reference/command-line-interface/#add
👍 1
I needed to explicitly name then in the yml for them to show?
As in 
capabilities
?
b
Oh great thank you, and yes. As in if my tap should be configured in some way where as it installs it automatically fills it into the yml instead of being manually added
r
Yeah, I think there was some discussion about a 
<tap> --about --format yaml
which could then be passed to 
meltano add --from-ref
, but that doesn't exist yet. I would recommend doing
Copy code
meltano add --from-ref <https://raw.githubusercontent.com/meltano/hub/refs/heads/main/_data/meltano/extractors/tap-canvas/mwcooley99.yml>
and then manually changing the 
pip_url
to 
-e <local path>
.
ℹ️ 1
b
Thank you Reuben this is helpful, I appreciate your assistance in this matter
r
No problem, happy to help 🙂
👍 1
e
As in if my tap should be configured in some way where as it installs it automatically fills it into the yml instead of being manually added
Regarding this, I had some thoughts about it a few weeks ago. I hope to get back to it soon and work on a PR 🙂 https://github.com/meltano/meltano/discussions/9349
👀 1
❤️ 1
4 Views
Open in Slack
PreviousNext
Powered by