https://linen.dev logo
Join Slack
Powered by
Try to collect all fields with `tap-github`, which...
# plugins-general
j
Try to collect all fields with 
tap-github
, which I collected with my custom tooling. config:
Copy code
config:
    plugins:
      extractors:
      - name: tap-github
        select:
        - commits.sha
        - commits.parents
        - commits.html_url
        # Does not work
        - commits.author.name
        - commits.pr_id
        - commits.pr_number
        - commits.stats
      ....
I use 
target-jsonl
to debug it. Issues: • The collected JSON does not contain fields below 
Does not work
comment above. The similar situation is in the case of pull_requests. E.g. I get 
"author": {},
• Don't know how to collect organization users - /orgs/{org}/members • Don't know how to collect repositories - /orgs/{org}/repos
OK, it seems that org/repo level could be crawled by 
tap-github-org-projects
. But installation fails:
Copy code
$ meltano add extractor tap-github-org-projects

Added extractor 'tap-github-org-projects' to your Meltano project
Variant:	rossmcdonald (default)
Repository:	<https://github.com/rossmcdonald/tap-github-org-projects>
Documentation:	<https://hub.meltano.com/extractors/tap-github-org-projects--rossmcdonald>

Installing extractor 'tap-github-org-projects'...
Extractor 'tap-github-org-projects' could not be installed: failed to install plugin 'tap-github-org-projects'.
  Running command git clone --filter=blob:none --quiet <https://github.com/rossmcdonald/tap-github-org-projects.git> /tmp/pip-req-build-1qifdmex
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [10 lines of output]
      running egg_info
      creating /tmp/pip-pip-egg-info-dm1j78z4/tap_github.egg-info
      writing /tmp/pip-pip-egg-info-dm1j78z4/tap_github.egg-info/PKG-INFO
      writing dependency_links to /tmp/pip-pip-egg-info-dm1j78z4/tap_github.egg-info/dependency_links.txt
      writing entry points to /tmp/pip-pip-egg-info-dm1j78z4/tap_github.egg-info/entry_points.txt
      writing requirements to /tmp/pip-pip-egg-info-dm1j78z4/tap_github.egg-info/requires.txt
      writing top-level names to /tmp/pip-pip-egg-info-dm1j78z4/tap_github.egg-info/top_level.txt
      writing manifest file '/tmp/pip-pip-egg-info-dm1j78z4/tap_github.egg-info/SOURCES.txt'
      file tap_github.py (for module tap_github) not found
      error: package directory 'tap_github' does not exist
      [end of output]
s
Hey @edgar_ramirez_mondragon @aaronsteers @pat_nadolny. It looks to me this should be possible using just the tap-github shouldn't it? Maybe you can help @jan_soubusta figure out the problem here (the non-working selectors)?
j
Open in Slack
PreviousNext
Powered by