Quite recently I was just wondering idly: If I understand correctly, the core abstractions that meltano provides over singer-io are 1. state management 2. plugin isolation theoretically, i think it would be possible for plugins to be written in languages other than python for cases where higher performance is needed. Its often difficult to identify where the bottlenecks truly lie. It may be the case that most pipelines are bottlenecked by the piping operation from tap to target. But in some cases where the tap fetches data and does transformation record by record, before piping to the target, then we might realize performance gains from writing the same tool in a language other than python. If meltano is able to manage plugins in a different language, then I think meltano may be suitable for jobs requiring higher performance. what do you all think?

If this MSSQL Tap from singer.io uses Clojure and works then I believe what you are looking to do could be possible.

nice!

Hey @haleemur_ali! If you write a plugin in Rust, it's very easy to package it as a Python application with something like maturin to make it pip-installable. See for example: https://github.com/edgarrmondragon/singer-rust/tree/main/singer_summarize. You also don't need to rely on

pip_url

. If you got a plugin written in any language, you can just point the plugin's

executable

to its location :)

that's pretty nice

I had a question around this. If we are writing in a different language like Java/Rust, we need to take care of the specification ourselves as SDK is not available. Is that understanding, correct?

I believe that's correct @Reeni Mathew