The surprising thing is that if you selected a nested property, it also selects all of its parents which was an unexpected behavior to me. For example, if I select

supplier.custom-fields.freight-terms.name

I get the following:

Selected properties:
        [selected   ] supplier.custom-fields
        [selected   ] supplier.custom-fields.freight-terms
        [selected   ] supplier.custom-fields.freight-terms.name
        [automatic  ] supplier.id
        [automatic  ] supplier.updated-at

which isn't great because then you can get a result like this: (my next problem to solve)

{"id": 2, "updated-at": "2024-11-01T11:05:35-05:00", "custom-fields__freight-terms__name": "COLLECT"}
{"id": 3, "updated-at": "2024-11-01T11:05:42-05:00", "custom-fields__freight-terms": null}
{"id": 4, "updated-at": "2024-11-01T11:10:13-05:00", "custom-fields__freight-terms": null}
{"id": 5, "updated-at": "2024-08-20T12:30:10-05:00", "custom-fields__freight-terms__name": "COLLECT"}

Trading people time right now for compute time by not just selecting everything, that's my opinion anyway

The surprising thing is that if you selected a nested property, it also selects all of its parents

I recall someone going over how to do this a while back

> I recall someone going over how to do this a while back I'll scan around and see if I can't find it in an archive somewhere! I think the parents getting selected is probably necessary if you are not flattening the schema, otherwise Meltano will just pop off the parent which takes the child with it. I haven't looked at the code to verify this. Really only an issue when flattening because it would result in the target table having the whole hierarchy as columns, haha!

In particular, flattening when the API doesn't return the child properties if the parent property is NULL, such as Coupa API.

@visch Not that this is vital information to you but I figured I would come back and just share my findings. I wasn't able to find anything on removing the parents from the selection so I just went ahead and did some manual updates to

singer_sdk

to force that behavior. It did nothing helpful because the child just gets deselected at one point or another if the the parents aren't selected. I ended up writing my own custom implementation of the

_flatten_record()

function in

singer_sdk/helpers/_flattening.py

. I added a condition (

if new_key in _flattened_schema_._get_("properties", {}):

) to ignore leaf node keys that aren't actually in the flattened schema.

def _flatten_record(
    record_node: t.MutableMapping[t.Any, t.Any],
    *,
    flattened_schema: dict | None = None,
    parent_key: list[str] | None = None,
    separator: str = "__",
    level: int = 0,
    max_level: int = 0,
    max_key_length: int = DEFAULT_MAX_KEY_LENGTH,
) -> dict:
    """This recursive function flattens the record node.

    The current invocation is expected to be at `level` and will continue recursively
    until the provided `max_level` is reached.

    Args:
        record_node: The record node to flatten.
        flattened_schema: The already flattened full schema for the record.
        parent_key: The parent's key, provided as a list of node names.
        separator: The string to use when concatenating key names.
        level: The current recursion level (zero-based).
        max_level: The max recursion level (zero-based, exclusive).
        max_key_length: The maximum length of the key. Defaults to 255.

    Returns:
        A flattened version of the provided node.
    """
    if parent_key is None:
        parent_key = []
    items: list[tuple[str, t.Any]] = []
    for k, v in record_node.items():
        new_key = flatten_key(k, parent_key, separator, max_key_length=max_key_length)
        # If the value is a dictionary, and the key is not in the schema, and the
        # level is less than the max level, then we should continue to flatten.
        if (
            isinstance(v, collections.abc.MutableMapping)
            and flattened_schema
            and new_key not in flattened_schema.get("properties", {})
            and (level < max_level)
        ):
            items.extend(
                _flatten_record(
                    v,
                    flattened_schema=flattened_schema,
                    parent_key=[*parent_key, k],
                    separator=separator,
                    level=level + 1,
                    max_level=max_level,
                    max_key_length=max_key_length,
                ).items(),
            )
        else:
          if new_key in flattened_schema.get("properties", {}):
            items.append(
                (
                    new_key,
                    serialize_json(v)
                    if _should_jsondump_value(k, v, flattened_schema)
                    else v,
                ),
            )

    return dict(items)

@Edgar Ramírez (Arch.dev) I am also curious on your opinion on this. Do you think it is a bug in the flattening logic that these properties not in the flattened schema are being retained? Or is this simply an edge case that should be handled via the custom Tap?

I don't have the 30-60 min right now 😕

I'd never expect others to take time to answer my questions anyways. I just like to throw them out there!