GitHub
06/05/2025, 5:50 PMcommits.* stream
version: 1
default_environment: dev
project_id: 32ff6ca8-5729-44ca-99b1-c62dad5a4ad7
environments:
- name: dev
- name: staging
- name: prod
plugins:
extractors:
- name: tap-github
variant: meltanolabs
pip_url: git+https://github.com/MeltanoLabs/tap-github.git
config:
repositories:
- AutoIDM/target-apprise
start_date: '2024-03-26T00:00:00'
select:
- commits.*
1. You'll get ~38 state messages because commits is a child stream of repository , and repository is a parent to a bunch of other streams for github. Even though they aren't selected they are still printed
Stack trace run through:
sdk/singer_sdk/tap_base.py
Line 468 in</meltano/sdk/commit/acb74875542af9f077e6395b17aa7237c2648975|acb7487>
| stream.finalize_state_progress_markers() |
| ---------------------------------------- |
(finalize_state_progres_markers ) is called each child_stream gets looped through for the repository stream.
sdk/singer_sdk/streams/core.py
Line 1001 in</meltano/sdk/commit/acb74875542af9f077e6395b17aa7237c2648975|acb7487>
| child_stream.finalize_state_progress_markers() |
| ---------------------------------------------- |
so all of the streams that are descendants of repository get their state written even though they aren't selected. Maybe this is a bug in the SDK? We could also check to be sure the stream is selected here
sdk/singer_sdk/streams/core.py
Line 1000 in</meltano/sdk/commit/acb74875542af9f077e6395b17aa7237c2648975|acb7487>
| for child_stream in self.child_streams or []: |
| --------------------------------------------- |
instead of just printing them all, but maybe there's some reason for it (I don't think so?)
Possible solutions
1. We could not add streams that aren't selected to parent streams see logs below
2. We could check to be sure the stream is selected here
sdk/singer_sdk/streams/core.py
Line 1000 in</meltano/sdk/commit/acb74875542af9f077e6395b17aa7237c2648975|acb7487>
| for child_stream in self.child_streams or []: |
| --------------------------------------------- |
### Code
Log example
```
visch@DESKTOP-9BDPA9T:~/git/meltano-projects/github_2_postgres$ meltano run tap-github target-postgres
2024-03-28T133830.011886Z [info ] Environment 'dev' is active
2024-03-28T133830.057801Z [warning ] kind: password is deprecated for setting definitions in favour of sensitive: true, and is currently in use by the following settings of extractor 'tap-github': 'auth_token'. Please open an issue or pull request to update the plugin definition on Meltano Hub at https://github.com/meltano/hub/blob/main/_data/meltano/extractors/tap-github/meltanolabs.yml.
2024-03-28T133830.060385Z [warning ] kind: password is deprecated for setting definitions in favour of sensitive: true, and is currently in use by the following settings of loader 'target-postgres': 'password', 'ssh_tunnel.private_key', 'ssh_tunnel.private_key_password', 'ssl_client_private_key'. Please open an issue or pull request to update the plugin definition on Meltano Hub at https://github.com/meltano/hub/blob/main/_data/meltano/loaders/target-postgres/meltanolabs.yml.
2024-03-28T133831.054001Z [info ] 2024-03-28 093831,053 | WARNING | tap-github | The stream 'stargazers' might conflict with previous implementation. Looking for the older version? Use 'stargazers_rest'. cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github
2024-03-28T133831.054409Z [info ] 2024-03-28 093831,053 | WARNING | tap-github | The stream 'stargazers_rest' is deprecated. Please use the Graphql version instead: 'stargazers'. cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github
2024-03-28T133831.055051Z [info ] 2024-03-28 093831,054 | INFO | tap-github | Added 'anonymous_contributors' as child stream to 'repositories' cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github
2024-03-28T133831.055291Z [info ] 2024-03-28 093831,055 | INFO | tap-github | Added 'assignees' as child stream to 'repositories' cmd_type=elb consumer=False name=tap-github producer=True
stdio=stderr string_id=tap-github
2024-03-28T133831.055510Z [info ] 2024-03-28 093831,055 | INFO | tap-github | Added 'collaborators' as child stream to 'repositories' cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github
2024-03-28T133831.055725Z [info ] 2024-03-28 093831,055 | INFO | tap-github | Added 'commit_comments' as child stream to 'repositories' cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github
2024-03-28T133831.055916Z [info ] 2024-03-28 093831,055 | INFO | tap-github | Added 'commits' as child stream to 'repositories' cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github
2024-03-28T133831.056148Z [info ] 2024-03-28 093831,055 | INFO | tap-github | Added 'community_profile' as child stream to 'repositories' cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github
2024-03-28T133831.056340Z [info ] 2024-03-28 093831,055 | INFO | tap-github | Added 'contributors' as child stream to 'repositories' cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github
2024-03-28T133831.056530Z [info ] 2024-03-28 093831,055 | INFO | tap-github | Added 'dependencies' as child stream to 'repositories' cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github
2024-03-28T133831.056748Z [info ] 2024-03-28 093831,055 | INFO | tap-github | Added 'dependents' as child stream to 'repositories' cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github
2024-03-28T133831.056937Z [info ] 2024-03-28 093831,055 | INFO | tap-github | Added 'events' as child stream to 'repositories' cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github
2024-03-28T133831.057126Z [info ] 2024-03-28 093831,055 | INFO | tap-github | Added 'issue_comments' as child stream to 'repositories' cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github
2024-03-28T133831.057316Z [info ] 2024-03-28 093831,055 | INFO | tap-github | Added 'issue_events' as child stream to 'repositories' cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github
2024-03-28T133831.057504Z [info ] 2024-03-28 093831,055 | INFO | tap-github | Added 'issues' as child stream to 'repositories' cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github
2024-03-28T133831.057713Z [info ] 2024-03-28 093831,055 | INFO | tap-github | Added 'labels' as child stream to 'repositories' cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github
2024-03-28T133831.…
meltano/sdkGitHub
06/05/2025, 5:50 PM