Hey all! First of all, thanks for all the effort put into the singer sdk, it's a joy to work with! I'm currently developing a tap connecting to an API endpoint that returns records in which the most appropriate replication key ("datetime") is found nested inside another property, like so:

{
  "type": "event",
  "id": "string",
  "attributes": {
    "metric": "string",
    "description": "string",
    "timestamp": "string",
    "properties": {},
    "datetime": "string"
  },
  "title": "string",
  "meta": {}
}

I'm not sure how to best get at the "datetime" property inside "attributes". Glancing over the methods inside the sdk it looks like replication keys can only be a top-level property (and maybe the singer spec explicitly states that too?). This is the structure I'm dealt with though so if anybody has any ideas, I'm all ears! 😅 Thanks!

If modifying you stream schema is an option, you might be able to bump

attributes.timestamp

to a top-level property using

post_process

and then set

replication_key

to that new property key. Not tested, but something like:

class TapStream(RESTStream):
    replication_key = "timestamp"

    // ...

    def post_process(self, row: dict, context: Optional[dict] = None) -> Optional[dict]:
        row[self.replication_key] = row["attributes"][self.replication_key]
        return row

Ditto what @Reuben (Matatika) said. It's best practice to promote incremental keys and primary keys top-level properties - and

post_process()

is a quick and easy way to do that. I'll add: 1. You can also remove/pop the attribute from the original location. (

row[self.replication_key] = row["attributes"].pop(self.replication_key)

) 2. You'll want to make sure whatever

schema

you define in the stream matches the updated version, not the original.

Such a great solution! Tested it out and looks like this will work like a charm. 🥳 Thank you so much for taking your time to help me out @Reuben (Matatika) And thanks for the additional comments @aaronsteers, those steered me in the right direction w regards to the schema