Hello, I'm a little bit confused with usage of ``select_filter`` extra for tap-postgresql. I'm trying to filter output of the tap based on the value of certain field (not primary key) in the source table. Is it even possible?

Is it even possible?

Yeah, but not using

select_filter

, but rather with stream maps.

Oh, great, thanks! I tried using

___filter___

and stumbled on next problem - replication key processing. Whole task I'm trying to solve is: I have table A in postgres with column

is_ready

than marks that row it ready to be synched. This table contain 7m+ rows, only 100+ rows are created/updated between syncs. Row is created with

is_ready=false

, next period it's updated to

is_ready=true

. Number of rows makes me think that the only reasonable way to sync is incremental. AFAIK, using filter will ignore all

is_ready=false

rows on the moment row is created and skip it forever. Am I right?

Is there a column like

updated_at

that's incremented when

is_ready

is set to

true

?

Yes, there is.

Unless I'm missing something, I think you could use that as the replication key so you won't miss any records

I tried but failed because multiple row have same

updated_at

because they are updated together

Did the tap crash?

No, just produced invalid results. Sometimes records can be skipped because particular

updated_at

seen before.