Hello all, I am setting up data infra at my place. Recently came across Meltano, and considering it for ELT. I went through the Getting Started guide and did some tutorials. I am not able to figure out one basic requirement. Is it not possible to connect to multiple sources of the same type? I need to ingest data from two different Postgres DB (along with some other api sources). However, it seems to me that the "config" for an extractor is tied as one to one. I cannot figure out a way to add another set of configuration for the same postgres extractor. All the tutorials available only cover one source of each kind and move on. Can someone please share the relevant links with me? I even tried

meltano add extractor tap-postgres

for the second time, but as expected, it doesn't work. It tells me the extractor is aready intalled and moves on, which is correct. Thanks

@vikas_prasad Good question, and you've already found the current answer in that docs link! I recognize that that's hardly a satisfying solution, though, and as it happens, I'm working on a much better native solution to this problem as we speak: https://gitlab.com/meltano/meltano/-/issues/2390 (https://gitlab.com/meltano/meltano/-/merge_requests/1957)! This will let you run

meltano extractor tap-postgres-some-source --inherit-from=tap-postgres

to create multiple instances inheriting from the same base plugin. If you also define a plugin with

name: tap-postgres

in your

meltano.yml

(like the one you have today), all plugins with

inherit_from: tap-postgres

will even inherit their parent's configuration so you can store shared config in once place 🙂

thanks for the quick response @douwe_maan. I must say, I am liking Meltano so far quite much for its modular design 👌 (I tried three to four other projects) Was a little surprised on realizing that there's no native support for this already. This seems like a day0 need. Anyway, this^ is the natural solution to solve for this problem. How close are we to get it out? Although, I can get myself moving by using "pipeline specific configuration", but this^ looks much cleaner. If it's gonna see the light of the day in the near future, I shall plan accordingly. If there are independent bits of it, where I could contribute to fast-track it, do let know 🙌

@vikas_prasad The feature works perfectly in that branch, I'm just putting the final touches to the tests and docs, hoping to release this on Monday 🙂