Hey guys, is it possible to iterate over the result of another stream? My case: 1. I get a customer list (REST-API). There are no addresses inside the result. 2. To get the addresses of a customer I have to call the specific customer as a single api call. For example: I get a list with 3 Customers (id's = 1,2,3) as one api call. To get the addresses I need to do three more calls with customer/1, customer/2, customer/3. 3. Is it possible to iterate over the id's (with several api calls) or is there another way? Thank you ! Greetings Nick

Hey Nick 👋 This sounds like a good use-case for hierarchical streams, which is currently on the backlog. You may also be able to use partitions (available already), though it may depend on how many customer entities you are fetching. With partitioning, each customer id will have its own key in the STATE message, which can get pretty huge). I haven't actually used partitions yet, but maybe @aaronsteers can say more 🙂

If you'd like to use partitions, you can use the

get_writeable_state_dict

 / 

parent_stream_types

 trick we use here as an example: https://gitlab.com/meltano/singer-sdk/-/blob/main/singer_sdk/samples/sample_tap_gitlab/gitlab_rest_streams.py#L131-199. In your case, you would have 

CustomersStream

 create a new partition

{customer_id: X}

on

AddressesStream

for each customer, and then

AddressesStream

would make a request to

customers/{partition['customer_id']}

to get each customer's address. As Ken mentioned, the downside is that all of those partitions would be stored in your pipeline's state file, so one per customer. Depending on the number of customers, that may be fine, or not.

@ken_payne and @douwe_maan are exactly right. I think hierarchical streams is the best solution. If customers are 1:1 with addresses and you want to store the address along with the customer, there's one additional option of using

post_process()

to manage your own additional rest call(s) to supplement the record data before passing it downstream. We don't have supported helper methods yet for those supplemental/complementary rest calls but that's an option you could consider as well.

Hey guys, sorry for my late reply. Thanks for your anwsers. I will have a look at your possibilities. Otherwise it really sounds like hierarchical streaming. Have a nice week! Greetings Nick