Question: Is the Gitlab Tap able to pull more than...
# troubleshootingc
carolus_holman
11/03/2021, 2:28 PMQuestion: Is the Gitlab Tap able to pull more than 100 projects? I am not seeing all the projects in my GitLab group. Thanks.
a
aaronsteers
11/03/2021, 2:50 PM@douwe_maan, @taylor - Do either of you know? @carolus_holman - I don't know this tap intimately but to confirm - are you seeing exactly 100 repos being pulled? This sounds suspiciously like the symptom of omitted pagination or a bug in the pagination logic.
c
carolus_holman
11/03/2021, 3:39 PMI tried a few things, I have 2 gitlab instances (Migration) from the limit seems to be 100. When I look at the Group API, it has all of the projects available, tried a few other things (pagination:keyset, order_by: id etc.), but can't seem to get around the 100 item limit... thanks.
carolus_holman
11/03/2021, 7:50 PMWell it seems that the code block I am trying to check is not being executed:
carolus_holman
11/03/2021, 7:50 PM# X-Total-Pages header is not always available since GitLab 11.8# <https://docs.gitlab.com/ee/api/#other-pagination-headers># X-Next-Page to check if there is another page available and iteratenext_page = 1_try_:_while_ next_page:# handle endpoints that return a single JSON object_if_ isinstance(resp_json, dict):_yield_ resp_json# handle endpoints that return an array of JSON objects_else_:_for_ row _in_ resp_json:_yield_ row_except_ ResourceInaccessible _as_ exc:# Don't halt execution if a Resource is Inaccessible# Just skip it and continue with the rest of the extraction_return_ []carolus_holman
11/03/2021, 7:51 PMI was thinking that the header was case sensitive (X-Next-Page) vs x-next-page.
carolus_holman
11/03/2021, 7:51 PMI have gone about a far as my python skillz will let me... 🙂
carolus_holman
11/03/2021, 7:53 PMLocated: .meltano/extractors/tap-gitlab-charterlab/venv/lib/python3.9/site-packages/tap_gitlab/__init__.py
a
aaronsteers
11/03/2021, 9:34 PMThanks, @carolus_holman. I found this which might be related: https://github.com/MeltanoLabs/tap-gitlab/issues/56
aaronsteers
11/03/2021, 9:35 PMSide note: we're migrating those tap/target repos from Gitlab to Github, hence the weird cross linking.
aaronsteers
11/03/2021, 9:36 PMSince Gitlab 12.6, the max number of projects returned is hardcoded to 100. There is no pagination available.
c
carolus_holman
11/03/2021, 10:28 PMI was able to using a pagination query in the API query. In API V4 using projects?page=2&per_page=100.
a
aaronsteers
11/03/2021, 11:22 PMNice!
c
carolus_holman
11/04/2021, 11:20 AMThat was manually! Can't get it to work with the Tap... 😐