question on contexts in taps.. I saw the suggestion that it is for when you have like.. shards of a dataset.. I think I have that situation now.. I have an endpoint which I have to query with all permutations of the alphabet from 1 to 5 letters.. it then compiles a response for that set of letters and this takes about 24 hours to run... I see it running now but I have coded to run the calculation of the permutations and the entire set of multiple queries within the single "Tap" call>> Im wondering are there good examples of this use of context for queries? should I perhaps lean towards splitting up my tap by using these alphabet permutations as the "slice" of the datasets response? Basically I either hand squash.. or push to postgresql via python by hand (until just now when I started using my tap) and I get rid of duplicates myself.. but maybe I can start to just let the context calls send the raw response and postgresql handles the rest? Please excuse any ignorance.. I am sort of a noob on postgresql and the data engineering side but super happy to see my tap going along.. will wait and see how it goes , it takes about 24 hours to finish

There are a couple layers to this question - and it's a great topic. 1. Ignoring orchestration for a second, on the tap stream side, you can override

Streams.partitions

property to be the list of shards you'd like to be able to run as distinct contexts. That would get your tap keep separate STATE trackers for each partition, and to pass the partition dict as

context

to any method supporting it. 2. On the tap side also, you can accept a config option that's a list of one or more partitions to run. When omitted, presumably, you'd run all the partitions defined above. However, when set by the user, you'd run only the partition(s) requested. 3. On the orchestration side (for example, from Meltano), you would kick off any number of runners each with distinct partition config being passed. For instance, if there are 50 partitions, I could send a list of 5 partitions each to 10 invocations, or 10 partitions each to 5 invocations. You could also send 1 each partition to each of 50 invocations. 4. Lastly, you'd need a plan to manage state - either keeping separate state tracking for each of the 5 or 10 invocations you'd run - or else coming up with a more advanced way of merging back the states afterwards. Does this help at all? Would be a good discussion for #C01QS0RV78D also if you are free tomorrow.

absolutely nailed the question AJ.. I will dive into these things in the docs.. if you have one favorite tap in mind which exhibits these behaviors you’ve talked about I’d love to just get the name and I can review the source from there.. otherwise I will finish watching the aethena tap video later this week and come back to this .. but it sounds like a plan.

@emcp Awesome. Unfortunately I am not aware of any great sharding/slicing samples out in the wild. If you do build one, I'd love to showcase it as a sample. Also, like I said above, this would make a fantastic #C01QS0RV78D discussion if you wanted to join us for a live discussion - before or after you start building it. 🙂

I just finished work so.. need to go out and fetch some things today.. but let me make some progress to educate myself on your pointers and hopefully next office hours it can be really fruitful and I get the easy stuff sorted