Is there a good way to have multiple partitions in a stream? For instance, if I need an

id

and a

year

and it needs to process all

ids

for all

years

. I currently have it working by serializing each permutation as a JSON string, passing it along as a partition, and unpacking the partition from the context. Just thinking if that is the best approach.

Lots of discussions on this topic are available here and in the referenced issues https://meltano.slack.com/archives/C01PKLU5D1R/p1664421878665089

If years isn't dynamic and you don't need state for each combo I'd probably go with something static in the tap just because it's a cludge right now to make work. If you could up vote the issue in there it helps meltano prioritize! (It actually does!)

It will be dynamic based on the config start_date and current_date

Got it, yeah dynamic and static is bad wording from me. I think in the tap without partitioning makes the most sense as you don't have to track state for each combo. If partioning worked with child streams out of the box it'd probably make sense to do it there but it doesn't today

Ah, I ran into the issue with merging parent and context. The current tap I’m working on offers another pathway that gives a list of ids to get the JSON method to work, but could easily see this becoming a challenge on another tap.

It's doable but just a pain, if you could up vote that issue I"ll tag it somewhere one sec