What's the purpose of

_MAX_RECORD_AGE_IN_MINUTES

? Is there no way to configure and increase this? https://github.com/meltano/sdk/blob/main/singer_sdk/target_base.py#L53

Is there no way to configure and increase this?

Not at the moment. Do create an issue in the github repo 🙏

Will do. Do you know the context of why this was introduced? It seems to me like that you wouldn't normally want to mix newer records when the original query was at a specific point in time. It would just be confusing if newer records existed, but some records aren't up to date because they were already extracted prior. Maybe I'm misunderstanding what it's doing?

Oh, I think I see where the confusion is.

_MAX_RECORD_AGE_IN_MINUTES

is just a signal for flushing records from memory. So when a loader is processing records from the extractor, it usually doesn't insert every record it receives immediately, instead it holds a batch of records in memory and waits for a signal for storing the batch. The signal could be: • the loader will start to process a different stream, so process the current one to free-up memory and commit the current state of the pipeline • the batch has reached a certain size limit, so process it • the batch has been holding records for 5 minutes now, let's process those records so in case something goes wrong we have already committed some of the data.

So I commented on this a little here in the issue I created. https://github.com/meltano/sdk/issues/2692 Not sure if I'm reading that comment correctly or not, but it seems to imply that it is a performance penalty for a batch flush operation https://github.com/meltano/sdk/issues/1064#issuecomment-1299098653