<#2350 bug: Child stream states are written even w...
# meltano-repo-activity
g
#2350 bug: Child stream states are written even when the child isn't selected Issue created by visch ### Singer SDK Version 0.36.1 ### Is this a regression? • Yes ### Python Version 3.8 ### Bug scope Taps (catalog, state, etc.) ### Operating System Linux ### Description To replicate with a tap 1. Setup tap-github 2. Select only the
commits.*
stream
Copy code
version: 1
default_environment: dev
project_id: 32ff6ca8-5729-44ca-99b1-c62dad5a4ad7
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-github
    variant: meltanolabs
    pip_url: git+https://github.com/MeltanoLabs/tap-github.git
    config:
      repositories:
      - AutoIDM/target-apprise
      start_date: '2024-03-26T00:00:00'
    select:
      - commits.*
1. You'll get ~38 state messages because
commits
is a child stream of
repository
, and
repository
is a parent to a bunch of other streams for github. Even though they aren't selected they are still printed Stack trace run through: sdk/singer_sdk/tap_base.py Line 468 in</meltano/sdk/commit/acb74875542af9f077e6395b17aa7237c2648975|acb7487> | stream.finalize_state_progress_markers() | | ---------------------------------------- | (finalize_state_progres_markers ) is called each child_stream gets looped through for the repository stream. sdk/singer_sdk/streams/core.py Line 1001 in</meltano/sdk/commit/acb74875542af9f077e6395b17aa7237c2648975|acb7487> | child_stream.finalize_state_progress_markers() | | ---------------------------------------------- | so all of the streams that are descendants of repository get their state written even though they aren't selected. Maybe this is a bug in the SDK? We could also check to be sure the stream is selected here sdk/singer_sdk/streams/core.py Line 1000 in</meltano/sdk/commit/acb74875542af9f077e6395b17aa7237c2648975|acb7487> | for child_stream in self.child_streams or []: | | --------------------------------------------- | instead of just printing them all, but maybe there's some reason for it (I don't think so?) Possible solutions 1. We could not add streams that aren't selected to parent streams see logs below 2. We could check to be sure the stream is selected here sdk/singer_sdk/streams/core.py Line 1000 in</meltano/sdk/commit/acb74875542af9f077e6395b17aa7237c2648975|acb7487> | for child_stream in self.child_streams or []: | | --------------------------------------------- | ### Code Log example ``` visch@DESKTOP-9BDPA9T:~/git/meltano-projects/github_2_postgres$ meltano run tap-github target-postgres 2024-03-28T133830.011886Z [info ] Environment 'dev' is active 2024-03-28T133830.057801Z [warning ]
kind: password
is deprecated for setting definitions in favour of
sensitive: true
, and is currently in use by the following settings of extractor 'tap-github': 'auth_token'. Please open an issue or pull request to update the plugin definition on Meltano Hub at https://github.com/meltano/hub/blob/main/_data/meltano/extractors/tap-github/meltanolabs.yml. 2024-03-28T133830.060385Z [warning ]
kind: password
is deprecated for setting definitions in favour of
sensitive: true
, and is currently in use by the following settings of loader 'target-postgres': 'password', 'ssh_tunnel.private_key', 'ssh_tunnel.private_key_password', 'ssl_client_private_key'. Please open an issue or pull request to update the plugin definition on Meltano Hub at https://github.com/meltano/hub/blob/main/_data/meltano/loaders/target-postgres/meltanolabs.yml. 2024-03-28T133831.054001Z [info ] 2024-03-28 093831,053 | WARNING | tap-github | The stream 'stargazers' might conflict with previous implementation. Looking for the older version? Use 'stargazers_rest'. cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github 2024-03-28T133831.054409Z [info ] 2024-03-28 093831,053 | WARNING | tap-github | The stream 'stargazers_rest' is deprecated. Please use the Graphql version instead: 'stargazers'. cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github 2024-03-28T133831.055051Z [info ] 2024-03-28 093831,054 | INFO | tap-github | Added 'anonymous_contributors' as child stream to 'repositories' cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github 2024-03-28T133831.055291Z [info ] 2024-03-28 093831,055 | INFO | tap-github | Added 'assignees' as child stream to 'repositories' cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github 2024-03-28T133831.055510Z [info ] 2024-03-28 093831,055 | INFO | tap-github | Added 'collaborators' as child stream to 'repositories' cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github 2024-03-28T133831.055725Z [info ] 2024-03-28 093831,055 | INFO | tap-github | Added 'commit_comments' as child stream to 'repositories' cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github 2024-03-28T133831.055916Z [info ] 2024-03-28 093831,055 | INFO | tap-github | Added 'commits' as child stream to 'repositories' cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github 2024-03-28T133831.056148Z [info ] 2024-03-28 093831,055 | INFO | tap-github | Added 'community_profile' as child stream to 'repositories' cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github 2024-03-28T133831.056340Z [info ] 2024-03-28 093831,055 | INFO | tap-github | Added 'contributors' as child stream to 'repositories' cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github 2024-03-28T133831.056530Z [info ] 2024-03-28 093831,055 | INFO | tap-github | Added 'dependencies' as child stream to 'repositories' cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github 2024-03-28T133831.056748Z [info ] 2024-03-28 093831,055 | INFO | tap-github | Added 'dependents' as child stream to 'repositories' cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github 2024-03-28T133831.056937Z [info ] 2024-03-28 093831,055 | INFO | tap-github | Added 'events' as child stream to 'repositories' cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github 2024-03-28T133831.057126Z [info ] 2024-03-28 093831,055 | INFO | tap-github | Added 'issue_comments' as child stream to 'repositories' cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github 2024-03-28T133831.057316Z [info ] 2024-03-28 093831,055 | INFO | tap-github | Added 'issue_events' as child stream to 'repositories' cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github 2024-03-28T133831.057504Z [info ] 2024-03-28 093831,055 | INFO | tap-github | Added 'issues' as child stream to 'repositories' cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github 2024-03-28T133831.057713Z [info ] 2024-03-28 093831,055 | INFO | tap-github | Added 'labels' as child stream to 'repositories' cmd_type=elb consumer=False name=tap-github producer=True stdio=stderr string_id=tap-github 2024-03-28T133831.… meltano/sdk