https://linen.dev logo
Join Slack
Powered by
Hi everyone! I just discovered Meltano and it look...
# getting-started
a
Hi everyone! I just discovered Meltano and it looks gorgeous as an idea! I am trying to play with it and as I have plenty of repos on Github I wanted to try 
tap-github
but I hit some obstacles. When I try to add 
tap-github
extractor I am getting the following error:
Copy code
$ meltano --log-level=debug add extractor tap-github
[...]
  File "/Users/adambabik/meltano-projects/.venv/lib/python3.9/site-packages/meltano/core/plugin_discovery_service.py", line 269, in get_base_plugin
    plugin = project_plugin.custom_definition or self.find_definition(
  File "/Users/adambabik/meltano-projects/.venv/lib/python3.9/site-packages/meltano/core/plugin_discovery_service.py", line 248, in find_definition
    raise PluginNotFoundError(PluginRef(plugin_type, plugin_name)) from err
meltano.core.plugin.error.PluginNotFoundError: Extractor 'tap-github' is not known to Meltano

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/adambabik/meltano-projects/.venv/lib/python3.9/site-packages/meltano/cli/__init__.py", line 51, in main
    raise CliError(str(err)) from err
meltano.cli.utils.CliError: Extractor 'tap-github' is not known to Meltano
Extractor 'tap-github' is not known to Meltano
It works fine with 
tap-gitlab
. The version I use is 
meltano, version 1.67.0
. I am on macOS.
d
Can you share your 
meltano.yml
?
t
I believe this is b/c tap-github isn't actually in a released version! https://gitlab.com/meltano/meltano/-/blob/master/CHANGELOG.md
For now you should be able to add it as a custom extractor
Sorry for the confusion on that - we'll get it released next week! cc @aaronsteers @edgar_ramirez_mondragon
a
Thanks @taylor! Actually, I tried this also on my Linux machine and I can install 1.77 there which contains tap-github.
t
Really? 🤨
a
Yeah, that's what 
meltano --version
is reporting
d
Ah yes, that may be it! Are you on Python 3.9 on your Mac? That will install an older version that didn’t list 3.9 as unsupported yet. Since 3.9 is unsupported, using 3.8 will get you the latest (1.77)
a
version 
1.77.0
yeah, I am on 3.9 on mac
and on 3.8 on Linux so that makes sense 🙂
BTW thanks so prompt responses!
and as we speak I have one more problem on linux with 
tap-github
. It seems like it downloads nothing... But there is no error, almost no logs
d
Can you share the output of 
meltano install
, even if it isn’t very informative?
a
Copy code
➜ meltano elt tap-github target-jsonl --job_id=github-to-jsonl --full-refresh -f
meltano      | Running extract & load...
meltano      | Performing full refresh, ignoring state left behind by any previous runs.
tap-github   | INFO Starting sync of repository: singer-io/tap-github
meltano      | Incremental state has been updated at 2021-07-09 19:40:31.281521.
meltano      | Extract & load complete!
meltano      | Transformation skipped.
this is from Linux so 1.77.0
ah sorry it's the job execution
d
OK that suggests it was installed correctly at least
a
Copy code
➜ meltano install
Installing 3 plugins...
Installing loader 'target-jsonl'...
Installing extractor 'tap-gitlab'...
Installing extractor 'tap-github'...
Installed loader 'target-jsonl'

Installed extractor 'tap-github'

Installed extractor 'tap-gitlab'

Installed 3/3 plugins
right it installs fine
d
I’m guessing nothing is showing up in your target’s destination? Can you share the 
meltano.yml
definition for 
tap-github
, specifically the 
capabilities
?
a
exactly, I follow the instruction from the main page but changed it for 
tap-github
. There is nothing in 
output/
here is 
meltano.yml
from the project root dir:
Copy code
➜ cat meltano.yml
version: 1
send_anonymous_usage_stats: true
project_id: 70b20e31-b001-4c01-98ef-ef520706836b
plugins:
  extractors:
  - name: tap-github
    variant: singer-io
    pip_url: git+<https://github.com/singer-io/tap-github.git>
    config:
      start_date: '2021-01-01T00:00:00Z'
      repository: singer-io/tap-github
    select:
    - commits.*
  - name: tap-gitlab
    variant: meltano
    pip_url: git+<https://gitlab.com/meltano/tap-gitlab.git>
    config:
      projects: meltano/meltano
      start_date: '2020-05-01T00:00:00Z'
    select:
    - tags.*
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl
d
OK, that looks good
Try running with 
meltano --log-level=debug elt …
to see if any messages are making it from the tap to the target at all
a
it's enormous output, anything particular I should look for?
d
Lines starting with 
tap-github (out)
a
there is only 
{"type": "STATE", "value": {}}
d
Hmm, OK, so the tap is not actually finding any records
Perhaps an 
access_token
is required, although I’d expect it to raise an error if it was required and not provided
a
exactly, and I do provide it via 
TAP_GITHUB_ACCESS_TOKEN
d
What does 
meltano select --list tap-github
show?
a
Copy code
➜ meltano select --list tap-github                                                                
Legend:
selected
excluded
automatic

Enabled patterns:
commits.*

Selected attributes:
[selected ] commits._sdc_repository
[selected ] commits.comments_url
[selected ] commits.commit
[automatic] commits.commit.author
[automatic] commits.commit.author.date
[automatic] commits.commit.author.email
[automatic] commits.commit.author.login
[automatic] commits.commit.author.name
[automatic] commits.commit.comment_count
[automatic] commits.commit.committer
[automatic] commits.commit.committer.date
[automatic] commits.commit.committer.email
[automatic] commits.commit.committer.login
[automatic] commits.commit.committer.name
[automatic] commits.commit.message
[automatic] commits.commit.tree
[automatic] commits.commit.tree.sha
[automatic] commits.commit.tree.url
[automatic] commits.commit.url
[selected ] commits.files
[selected ] commits.html_url
[selected ] commits.id
[selected ] commits.parents
[selected ] commits.pr_id
[selected ] commits.pr_number
[automatic] commits.sha
[selected ] commits.url
d
OK, that all looks good
You’re not seeing a 
SCHEMA
message written for the 
commits
stream by this line? https://github.com/singer-io/tap-github/blob/master/tap_github/__init__.py#L1065 That would have that 
tap-github (out)
prefix in the output
Aah I think I know what’s up
tap-github
should have 
properties
under 
capabilities
, not 
catalog
/cc @taylor
We can confirm that if you replace the 
name
, 
variant
, and 
pip_url
in your 
tap-github
definition in 
meltano.yml
with this entire snippet: https://gitlab.com/meltano/meltano/blob/master/src/meltano/core/bundle/discovery.yml#L670-698, and you then change 
catalog
to 
properties
under 
capabilities
Leave the 
config
and 
select
you have in 
meltano.yml
right now
@taylor If that works, please make that change in 
discovery.yml
, and double-check the same thing for all other taps we added recently!
a
ok, let me try that
t
Yep, I'll make an issue
a
And it works 🙂 thanks!
d
yay!
a
awesome, I can play with loaders now 🙂
t
https://gitlab.com/meltano/meltano/-/merge_requests/2209 should be the fix for the github and jira definitions. Sorry you ran into that @adam_babik!
d
@adam_babik Once that MR goes in, you can revert your 
meltano.yml
definition to only `name`/`variant`/`pip_url` again!
a
Perfect!
Open in Slack
PreviousNext
Powered by