Chen
03/06/2024, 11:04 AMversion: 1
default_environment: dev
project_id: 387e5070-1c5f-4920-80c5-0ec1122554aa
environments:
- name: dev
plugins:
extractors:
- name: tap-gitlab
variant: meltanolabs
pip_url: git+<https://github.com/MeltanoLabs/tap-gitlab.git>
config:
start_date: '2024-04-01T00:00:00Z'
api_url: <https://jihulab.com>
groups: group8270010
projects: my-test-project
select_filter:
- commits
loaders:
- name: target-csv
variant: hotgluexyz
pip_url: git+<https://github.com/hotgluexyz/target-csv.git>
config:
destination_path: result
this is my meltano.yml, Adding the select part will prompt that there is an error in the configuration file. I think my configuration should be correct, at least it looks the same as the example given by the community.
Can anyone help me figure out where the problem is? I just want to get some commit information from gitlab. If I don't add the select field, all users will be fetched from /v4/users. I don't want to get it. Because the data volume is too large, I only want to get the specified information in a certain project, such as commit, project_id, commit_id
I uploaded a screenshot of the program running. From the screenshot, you can see the accumulated time collected by Meltano when acquiring users. My question is how can I obtain only specific information of a certain project?Edgar Ramírez (Arch.dev)
03/06/2024, 4:01 PMAdding the select part will prompt that there is an error in the configuration fileIt may be a yaml indentation error or something similar. Can you share the error message you see when you try to run after setting
select_filter
?Chen
03/08/2024, 6:43 AMEdgar Ramírez (Arch.dev)
03/08/2024, 2:58 PMplugins:
extractors:
- name: tap-gitlab
variant: meltanolabs
pip_url: git+<https://github.com/MeltanoLabs/tap-gitlab.git>
config:
projects: meltano/meltano
start_date: '2015-01-01'
select:
- "commits.*"
this should give you only the commits