is there a simple way to get a child stream to use...
# singer-tap-developmentd
david_wallace
09/02/2022, 6:27 PMis there a simple way to get a child stream to use a replication key that isnt partitioned by the child context? As in, all child stream partitions share the same replication key value?
david_wallace
09/02/2022, 6:32 PMI tried setting
state_partition_keys[] Copy code
{
"bookmarks": {
"merchants": {},
"transactions": {
"replication_key_signpost": "2022-09-02T18:31:50.388980+00:00",
"starting_replication_value": "2022-06-01",
"progress_markers": {
"Note": "Progress is not resumable if interrupted.",
"replication_key": "transaction_time",
"replication_key_value": "2022-08-31T15:17:02+00:00"
}
}
}
}e
edgar_ramirez_mondragon
09/02/2022, 8:00 PMHi @david_wallace ! Have you tried setting the
is_sortedTrued
david_wallace
09/02/2022, 8:01 PMI have! didnt seem to have an effect. my hypothesis is that this is due to the fact that we muck around with
contexte
edgar_ramirez_mondragon
09/02/2022, 9:51 PMmy hypothesis is that this is due to the fact that we muck around withAh yeah, that may be it. Issue’s probably around this line: https://github.com/meltano/sdk/blob/be8e0eb51a4e53dea790ef06dafa48c5067cda92/singer_sdk/streams/core.py#L1019during ingestion (we add some keys to it)context
d
david_wallace
09/02/2022, 9:55 PMbingo
david_wallace
09/02/2022, 9:57 PMfwiw, i find myself needing to pass some arbitrary extra info to stream methods quite often, and I just assumed thats what context was for. it would be really nice if there was some sort of facility for passing arbitrary metadata through the stream components
a
aaronsteers
09/02/2022, 10:16 PMi find myself needing to pass some arbitrary extra info to stream methods quite often, and I just assumed thats what context was for. it would be really nice if there was some sort of facility for passing arbitrary metadata through the stream componentsThis would make a really good feature proposal if you want to make an issue. Class-level and object-level declarations are tricky because the same class may be invoked several times, and the same stream object (at least as of now) may handle calls for multiple parents or partitions. All that said, it's certainly something that could help out developers, I think.
aaronsteers
09/02/2022, 10:18 PMTo the issue of stream state not being finalized - is this even after the tap finishing the other streams and shutting down normally (with success)?
aaronsteers
09/02/2022, 10:19 PMBefore the tap shuts down, it should finalize all steams' state objects.
https://github.com/meltano/sdk/blob/be8e0eb51a4e53dea790ef06dafa48c5067cda92/singer_sdk/streams/core.py#L1022-L1025
aaronsteers
09/02/2022, 10:20 PMIf the SDK is not finalizing states for progress_markers at the end of the Tap's execution (and all streams were successful), then that would probably be a bug.
d
david_wallace
09/02/2022, 11:15 PMThis would make a really good feature proposal if you want to make an issue. Class-level and object-level declarations are tricky because the same class may be invoked several times, and the same stream object (at least as of now) may handle calls for multiple parents or partitions.
All that said, it’s certainly something that could help out developers, I think.cool - I can def write up an issue this weekend @aaronsteers
david_wallace
09/02/2022, 11:16 PMTo the issue of stream state not being finalized - is this even after the tap finishing the other streams and shutting down normally (with success)?yep! this is after all streams finish successfully
david_wallace
09/02/2022, 11:17 PMcan try to collect some logs to show ya
a
aaronsteers
09/03/2022, 12:13 AM👍 Thanks!!