<#8807 bug: Selecting all nested properties with `...
# meltano-repo-activity
g
#8807 bug: Selecting all nested properties with
my_stream.prop.*
should imply selecting the top-level property
my_stream.prop
Issue created by edgarrmondragon ### Meltano Version 3.5.4 ### Python Version 3.12 ### Bug scope CLI (options, error messages, logging, etc.) ### Operating System NA ### Description Configuring
select
with: select: - my_stream.prop.* selects all nested sub-properties of
my_stream.prop
, but fails to select the parent property itself. This catalog output causes the SDK schema pruning algorithm to drop
prop
and all of its nested properties from the schema. The recommended workaround is to explicitly select `prop`: select: - my_stream.prop - my_stream.prop.* Ideally users would not need to do this and it's arguably bad UX that they need to. One way we might be able to address this is by generating
MetadataRule
instances for these special cases. We already generate two rules for a single pattern in some cases: meltano/src/meltano/core/plugin/singer/catalog.py Lines 188 to 208 in</meltano/meltano/commit/3b7a3f900644636ea44df3d5213aa2abe91d1859|3b7a3f9> | if selected or not prop_pattern or prop_pattern == "*": | | ------------------------------------------------------- | | rules.append( | | MetadataRule( | | tap_stream_id=pattern.stream_pattern, | | breadcrumb=[], | | key="selected", | | value=selected, | | ), | | ) | | | | if prop_pattern: | | props = prop_pattern.split(".") | | | | rules.append( | | MetadataRule( | | tap_stream_id=pattern.stream_pattern, | | breadcrumb=property_breadcrumb(props), | | key="selected", | | value=selected, | | ), | | ) | For example, the pattern
my_stream.prop.*
generates the rules MetadataRule("my_stream", [], "selected", value=True) MetadataRule("my_stream", ["properties", "prop", "properties", "*"], "selected", value=True) but we could patch so it also generates MetadataRule("my_stream", ["properties", "prop"], "selected", value=True) Related: • #3024 • meltano/sdk#2687 • https://gitlab.com/meltano/sdk/-/issues/285#note_750983833 ### Code No response ### Link to Slack/Linen No response meltano/meltano