I'd like to load only rows that are from >24hrs before "now" (whether that's run-start or time of query doesnt matter much to me). I'm trying to solve this with stream-maps, particularly the meltano-map-transform plugin. I thought that stream-maps would be able to solve this, but I didn't realize how limited the python eval is within a filter... there's no datetime library, and I dont think there are any datetime objects in there to reference. So I have an incoming date column, but cant seem to find a way to get a current timestamp, nor to compare two timestamps easily. I havent been able to find a solution to this yet, using stream-maps or any other strategy. I'm assuming I'd be able to solve by writing my own tap, but I'd like to avoid that if possible. I'm currently using the transferwise variants of the postgresql tap and snowflake target. Anyone have ideas? Is there any way to pull this off?

@quinn_batten - Can you say a bit more about your use case? Can I assume you want to do incremental loads from postgres, but you just don't want too-new records which have not "cooled off" in the DB?

It's a cheap hack but if you're using INCREMENTAL mode you can define a view that includes the date constraint and then replicate that view. I guess I don't know for sure that that'll work with a Postgres source but I know it works with a MySQL source. 😉

AJ, exactly— the table values can be updated for up to 24 hours after the row is first created. Thomas that's a good sol'n but unfortunately doesnt work for this case, as we cant alter that souce DB at all (it's read replica of the prod webapp db)

@quinn_batten - Thanks for confirming. I've added the

Accepting Pull Requests

label to the above, and fwiw, we have a very early version of an SDK-based

tap-postgres

here: https://github.com/meltanolabs/tap-postgres which could be used to test out the functionality.

Awesome, thanks AJ! Yess I've been watching that variant closely, super excited about it