How does SDK implement state feature? There are AP...
# singer-tap-development
r
How does SDK implement state feature? There are APIs that allow filtering through the HTTP request itself (filter through replication keys) and some APIs that return all data, How should one access a state object inside functions like
Copy code
prepare_request_payload()
for a tap in order get the bookmark values so they can add it in a payload for the API POST Request. I am able to successfully send the requests but i want the value of replication keys in the state file be used in the request as part of state feature .
a
Hi, @rithesh_s - For datetime-type replication keys, you can use
Stream.get_starting_timestamp(context)
, which takes the same context passed to
prepare_request_payload()
and returns a datetime value. For non-datetime types, use
Stream.get_starting_replication_key_value()
. The two are mostly identical except that the datetime version also considers the
start_date
config value, if set.
In general, if you use those two methods, you should be good. We don't want developers to be required to know all the internals for how state is managed, because there are a ton of edge cases, but if you are curious, and for debugging purposes, we do have additional details here: SDK Implementation Details - Stream State — Meltano SDK 0.2.0 documentation
I do feel like we should add more docs about those two methods. This will probably be improved in a future docs update.
r
Does the timestamp version return the replication value for a stream present in a state file if --state is used?
@aaronsteers
If state is not used it, then does SDK search for a the replication field value in a config file ?
a
Yes, that's correct. Timpestamp version will return the matching replication value when --state is passed and a matching stream entry exists within the state file, and if state is not used, the SDK will return
start_date
from config if set. Otherwise (if neither is true) would be null.
r
Thanks a lot
a
To help future inquiries, I've added everything from this thread to the docstrings: Improved docstrings for stream class (!105) · Merge requests · Meltano / Meltano SDK for Singer Taps and Targets · GitLab Once merged, these will be These are now rendered in the auto-generated docs at https://sdk.meltano.com/en/latest/classes/singer_sdk.Stream.html#singer_sdk.Stream.get_starting_replication_key_value Still more to do, probably, to make this page discoverable - but it's a start. 🙂
r
Great Work!!
a
Thanks! 🙂