Hi everyone! I'm working on ELTP (reverse ETL as some like to call it) use case where data is extracted from multiple SaaS-systems and then stored in a Snowflake warehouse. Within Snowflake we do a bunch of transformations using dbt. In the last step I'm trying to publish data back into the SaaS-systems using custom targets that make use of the SaaS-systems API's. I'm using the SDK to build the custom targets, yet this seems not to be as elaborate in terms of utilities compared to the developing experience for custom taps. For instance, regarding authentication with taps there are many different

Authenticator

classes that you can make use of which I did not see for targets. Also, the best way to couple streams to sinks is not very intuitive to me. Am I missing something or is it indeed the case that API-based custom targets have not been a focus for the SDK? And is this by any chance something that is on the roadmap? P.S. Any resources / best practices on how to set-up such API-based custom targets for SaaS systems would be welcome! (now mostly looked at https://gitlab.com/DouweM/investing-to-lunch-money)

Hi, @bernard_wezeman! You are right to call out the SDK's relatively less mature helper classes for API/SaaS targets. The corollary for the class RESTStream or GraphQLStream would be something like a RESTSink but as of now we only have RecordSink and BatchSink. Those classes are capable of supporting SaaS targets, but not really optimized for them. In theory, RESTSink could be created and added to the SDK, and if so, I imagine that would have the same kind of streamlining as RESTStream has (around the

requests

library specifically). Also, some RESTSinks would use

POST

for one more more records at a time, and some might use

PUT

for exactly one record at a time. Those could represent separate subclasses or just implementation options on the same class. 🤔 Hope this helps! cc @visch who has worked on some reverse ETL use cases, in case he has some thoughts on this...

I agree with @aaronsteers. I also think what we’re missing is more mileage with rest targets in the wild to figure out the right abstractions.

@aaronsteers @edgar_ramirez_mondragon Thanks for the feedback! I'll discuss it with my team and we'll workout a concept on how it would work nicely for our use case, while also trying to keep more general use cases in mind. We'll try to tag along in one of the office hours sessions in the upcoming weeks.

Thanks, @bernard_wezeman! Feel free to 'thumbs up', subscribe, and add comments to this new discussion in GitHub: Proposal discussion: Specialized `RESTSink` class to aid in writing SaaS targets · Discussion #854 · meltano/sdk (github.com)