Hello :wave: I’m familiarising myself with meltano...
# getting-started
s
Hello 👋 I’m familiarising myself with meltano and going through Tutorial Part 1 - Connect to Data . I’ve followed the instructions and get this error when I try to run the pipeline `meltano run tap-github target-jsonl`:
Copy code
2024-07-30T12:48:52.245405Z [info     ]     raise RuntimeError(        cmd_type=elb consumer=False job_name=dev:tap-github-to-target-jsonl name=tap-github producer=True run_id=90985761-8ce2-458f-9c8e-a88752e7bd16 stdio=stderr string_id=tap-github
2024-07-30T12:48:52.245457Z [info     ] RuntimeError: All GitHub tokens have hit their rate limit. Stopping here. cmd_type=elb consumer=False job_name=dev:tap-github-to-target-jsonl name=tap-github producer=True run_id=90985761-8ce2-458f-9c8e-a88752e7bd16 stdio=stderr string_id=tap-github
2024-07-30T12:48:52.322575Z [info     ] Incremental state has been updated at 2024-07-30 12:48:52.322564.
2024-07-30T12:48:52.329718Z [error    ] Extractor failed              
2024-07-30T12:48:52.329919Z [error    ] Block run completed.           block_type=ExtractLoadBlocks err=RunnerError('Extractor failed') exit_codes={<PluginType.EXTRACTORS: 'extractors'>: 1} set_number=0 success=False
I’ve checked my GitHub token and I can print commits without any problem using PyGitHub:
Copy code
from github import Github

# Authentication is defined via github.Auth
from github import Auth
from dotenv import load_dotenv
import os

load_dotenv()
GH_TOKEN = os.environ['TAP_GITHUB_AUTH_TOKEN']
print(GH_TOKEN)

# using an access token
auth = Auth.Token(GH_TOKEN)

# First create a Github instance:

# Public Web Github
g = Github(auth=auth)

g.get_user().login

# %%
repo = g.get_repo("SoumayaMauthoorMOJ/test")

commit = repo.get_commit(sha="b6a5cb8bc0d3be339de8c6f5f36183aa7dc055c5")

print(commit.commit.author.date)
Any idea what I’m doing wrong?
1
This is the copy of my meltano config file:
Copy code
version: 1
default_environment: dev
project_id: 9deeee6d-844d-449e-b36c-2be9c3f9fa64
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-github
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-github.git>
    config:
      start_date: '2024-05-01'
      repositories:
      - SoumayaMauthoorMOJ/test
    select:
    - commits.url
    - commits.sha
    - commits.commit_timestamp
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl
a
Were you able to extract any data via meltano?
https://github.com/MeltanoLabs/tap-github/issues/222#issuecomment-2025207055 Infact this link suggests you might get a message about rate limit even if your token is invalid.
s
meltano config tap-github test
gives me this error:
Copy code
Plugin configuration is invalid
RuntimeError: All GitHub tokens have hit their rate limit. Stopping here.
Sorry all sorted the
.env
file was in the root of my repo, not in the meltano project
👍 2