Hi everyone, I'm wondering if there is a version of

tap-github

built on the meltano singer sdk (I've searched hub.meltano but haven't found one)? I've been using https://github.com/singer-io/tap-github (outside of meltano) but found I had to make a lot of changes (update schema definitions, alter various parameters like page_size for api calls, etc...). Their code is not exactly the cleanest, so I'm wondering if another version might make those changes less clunky. Or if I should just bite the bullet and implement yet another variant built on your sdk...

AJ started working on one to pull data from GitHub to populate on the hub. https://github.com/dataops-tk/tap-github I don't think it pulls everything the singer-io one does, but perhaps you could extend it?

I was looking into this recently too @laurent, an sdk based tap would be great imo.

@taylor I just took a quick look at AJ's version, but didn't realise it was based on the sdk. if it is, I'd be happy to throw a bunch of PRs to add extra streams, at least the ones I need

I think that'd be worth it! @aaronsteers any thoughts?

@laurent - Honestly, that sounds great. And yes, this is on the SDK. Would love it to be expanded to cover more stream types!

I'm looking at it right now, I like what I'm seeing so far 🙂

For our use case, we've built the whole thing around "search" but you'd probably want to make that optional - and add a traditional

repo_list

,

org_list

or similar so it can more easily manage use cases where the repos are already known and enumerated.

though I might need a first PR to support python 3.9 😉

Just did the python 3.9 support bump here on another repo and it was effortless 🙂

possibly, let me first figure out if the code can help my use case. For this project, I will need a

tap-gitlab

soon as well, so it'd make sense to work on those code bases together, at least for me 🙂

Just jumping in to say I’m happy to help. I’m also using the existing

tap-github

and have had to make changes. I’ve been waiting for an excuse to get my hands dirty with the sdk!

Awesome! Thanks, @laurent for the PR (now merged! shipitparrot ). And thanks, @oliver_laslett - I think I do want to move that tap into MeltanoLabs and out from dataops-tk. Any objections to me doing that sooner than later? (I think Github/git will auto-redirect but worst case scenario, you might need to re-clone.)

@aaronsteers I've actually started on the second point above, which is my use case: get various streams for a given list of repos. I left a note on the issue as well. Thanks for merging the PR so quickly!

hey @laurent -- have you had success in being able to extract data from private repos? i just tested out the meltano labs fork and tried searching for a private repo to no avail, and i've been wrestling with some weird errors the last couple hours, also to no avail: https://meltano.slack.com/archives/C01TCRBBJD7/p1631129487118700