Hi , Can anyone suggest How do I use get_starting...
# meltano-plugin-development
h
Hi , Can anyone suggest How do I use get_starting_replication_value(context) In any method to use in below property @property def partitions (self): end_date= datetime.now() Start_date= #TODO read from state #TODO process dates return #processed date value partitions
e
The stream has a
context
attribute starting with singer-sdk 0.39.0
h
@Edgar Ramírez (Arch.dev) thanks for response! I have tried to access in the above defination as self.context but it is giving as None. I want to read replication key value from state in this definition.Your help will be greatly appreciated
e
What value do you expect it to have in this context? I'm also curious what you're trying to accomplish by reading the state to generate partitions. I've never done something like this, so I wonder if you'd end up with an exploding state: partitions are initially defined by a single replication value, then each partition (N) is synced and the state of each is stored.
h
The context i have been expecting is a stream context so that I can read my replication value...my state file looks exactly like this and i'm exploring the ways where I can read the replication value from state and do partitioning on date(like every 7days is a one partition from -N days from state value) and store very latest max date value in state
Screenshot_2024-08-23-21-50-08-05_40deb401b9ffe8e1df2f1cc5ba480b12.jpg
@Edgar Ramírez (Arch.dev) any suggestions on this?
e
So your state is already partitioned by
storeId
.
and store very latest max date value in state
This makes me thing you want to paginate over periods of 7 days. Maybe something like https://github.com/meltano/sdk/issues/2318?
ty 1