So I'm struggling on the best way to integrate a new API endpoint. Most of my endpoints just require a "fromdate" and "todate" and return records created or updated in that range. I have one of these that returns a bunch of "customer_id"s for instance and updates associated with those customer records - this is how I'm creating my dimensions. However, if I want the customer detail from the API I need to pass in a list of all the customer id's (as a parameter) I want detail for. I'm wondering the best way to do this extract. Right now I think I would either have to create a new stream in my custom tap (new class and new logic) and make the connection to snowflake in that new stream logic to get all the rows and customer_id's (from my customer dim table or maybe the stg table) that I need to pass through as a parameter to the API and do it that way (query them, store in memory, loop through and pass them as params), OR I could create a python dbt model that runs on snowflake so any queries to my tables would probably be faster - but then if I'm doing the extract outside of my tap, I'd have to recreate the auth/token logic. Any ideas?

https://meltano.slack.com/archives/C068YBV6KEK/p1732727984018779?thread_ts=1732717428.848659&cid=C068YBV6KEK and https://github.com/meltano/sdk/issues/2809 seem relevant.