Hi all, I’m doing some planning for next year and one of the major tasks on our radar is to better use the data in our warehouse in downstream places (CRM, ESP, etc.). This is the typical “reverse ETL” use case (though I know this channel doesn’t love that term - neither do I). Since we’re currently using Meltano for ingestion into the warehouse, it’d be nice to use it for publishing elsewhere, and I’m curious what you all might recommend on that front, either now or in the future based on planned roadmap elements. More specifics of what I’m imagining in thread.

Slightly clearer picture, basically explains what I'm doing. Left = Singer Sources Right I point to Singer Targets Using meltano of course 🙂

@nick_hamlin I’d say the big challenge for using Meltano for operationalizing data (Reverse ELT) is the existence of taps and targets for your use case. Paths 2 and 3 I think are the same from our perspective. Meltano and Singer would work very well for it. Some new functionality coming out very soon would make it easy to coordinate ELTEL type workflows as well (check out some of the meltano run work). @aaronsteers can you chime in on where the SDK is for SaaS targets?

@taylor yep, totally agree - I’d much prefer not to add another tool to the stack, but also recognize that having the right targets (both existing and maintained) is a major barrier to entry for options 2/3

@nick_hamlin a solution to that if you have budget would be to look at some of our consulting partners (shoutout to @visch) to help build those. I think you’d see a lot of interest from folks around this in the community 🙂

Thanks, those both seem like solid options too!

@nick_hamlin I previously used Singer for pushing data from snowflake to various other saas tools (Salesforce, Salesforce Marketing, Delighted, Amplitude, etc.). None of those targets were open sourced by my previous employer but we were able to get new targets up and running pretty quick using the SDK, its generally less complicated than taps since theres not really any pagination or bookmarking to do. Another Meltano feature that makes this way easier (which I wish we had back when I was doing this) is the stream maps which helps you rename and map tap streams to targets entities. Before this it was difficult to not put transformation logic in the targets or have to make a new snowflake view for every dataset to rename fields. I'm happy to share more details about my experience if you have questions - hope thats helpful

Thanks Pat, glad to hear that that’s a viable path as well!

I came here to add my comments but looks like @pat_nadolny and @taylor have got it covered. 🎉 I like "ELTP" as the acronym for an ELT pipeline with a final "publish" step but yes, as @pat_nadolny mentions: you can certainly use Singer for that if there's a target that matches the SaaS you want to land data into. The dev experience for creating a SaaS target primarily revolves around overriding RecordSink.process_record() for SaaS's that prefer singleton writes or BatchSink.process_batch() for SaaS targets are more efficient in batch.