Hi All - We are trying to get key-based replication working for tap-oracle but it seems that whilst the option exists in the tap, the replication doesn't actually work. We have set the replication key and we are using postgres to store the state (rather than sqllite). We can see that the state is being written to the run table and that the correct replication key (and value) is being referenced in the run table. However, when we run the pipeline we can see that every single record in the target db (postgres) is being updated, every single time we run the pipeline. This means that the key-based replication is not working as intended. We have gone through the tap-oracle (pipelinewise) code and we can't find where the state is read from the run table. We've also taken a look at the tap-postgres code and we can't find that functionality either. I am very confident that key-based replication must work for one or both of those taps so what are we missing? Any help would be much appreciated, thanks!

Hey @tom_saunders just checking, you got incremental replication right? The code you're looking for is inside https://github.com/transferwise/pipelinewise-tap-oracle/blob/master/tap_oracle/sync_strategies/incremental.py I think, (

stream_version = singer.get_bookmark(state, stream.tap_stream_id, 'version'))

. Or are you using something different?

That's right yeah, but as far as my colleague @meraj is aware, that method doesn't actually get the replication key value from the run table.

Hi @Sven Balnojan, Thanks for your response. This line of code _stream_version = singer.get_bookmark(state, stream.tap_stream_id, 'version'))_ is always returning None. In the same file line 58 (_*replication_key_value = singer.get_bookmark(state, stream.tap_stream_id, 'replication_key_value')*_) the replication key value is retrieved from the state and this is also returned as None. I think the problem is in https://github.com/transferwise/pipelinewise-tap-oracle/blob/master/tap_oracle/__init__.py. The '*replication_key_value*' in the state is not loaded from the state before calling the _state = incremental.sync_table(conn_config, stream, state, desired_columns)_,

mh, thanks for the insights, now I'm out of my depth - don't have a proper test setup yet and am not yet familiar with that code base. But for FWIW the key should be logged, so if it's None, it should either raise an exception or print out "replication key is None".

To continue trying to work this issue out I'll pose a different question. Has anyone ever seen incremental loads work with key-based replication for a sql-based tap? If so we can just look at the code for that tap and see how they've implemented it. Thanks.