Reuben (Matatika)
01/18/2025, 3:46 AMSQLStream
to create a "well-known" stream that defines a replication_key
that is respected (as in standard `RestStream`/`Stream` implementations)?
client.py
class TestStream(SQLStream):
connector_class = TestConnector
class AStream(TestStream):
name = "a"
replication_key = "updated_at"
class BStream(TestStream):
name = "b"
replication_key = "updated_at"
tap.py
KNOWN_STREAMS = {
"a": stream.AStream,
"b": streams.BStream,
}
...
def discover_streams(self):
for stream in super().discover_streams():
if stream_class := KNOWN_STREAMS.get(stream.name):
stream = stream_class(
tap=self,
catalog_entry=stream.catalog_entry,
connector=self.tap_connector,
)
yield stream
I've played around with this and a few other variations, but can't seem to get it to work - the sync runs as if no replication key was defined. Is there another (possibly simpler) way to do this?