Hello all I am setting up data infra at my place Recently ca

Question

Hello all I am setting up data infra at my place Recently came across Meltano and considering it for ELT I went through the Getting Started guide and did some tutorials I am not able to figure out one basic requirement Is it not possible to connect to multiple sources of the same type I need to ingest data from two different Postgres DB along with some other api sources However it seems to me that the config for an extractor is tied as one to one I cannot figure out a way to add another set of configuration for the same postgres extractor All the tutorials available only cover one source of each kind and move on Can someone please share the relevant links with me I even tried `meltano add extractor tap postgres` for the second time but as expected it doesn t work It tells me the extractor is aready intalled and moves on which is correct Thanks

bitter-alarm-89642 · Answer

I am using the following envs for the first postgres source in my ` env` ```export TAP POSTGRES HOST= export TAP POSTGRES PORT= export TAP POSTGRES DBNAME= export TAP POSTGRES USER= export TAP POSTGRES PASSWORD= export TAP POSTGRES FILTER SCHEMAS= export TAP POSTGRES DEFAULT REPLICATION METHOD= export TAP POSTGRES SELECT=``` seeing the naming convention I don t think there is a way to add another postgres source Unless there is a way to provide custom names for these env vars per source If it is indeed not possible I am a little surprised isn t it a very basic case

bitter-alarm-89642 · Answer

okay I think I found something <https meltano com docs integration html pipeline specific configuration>

bitter-alarm-89642 · Answer

let me try this and get back

ripe-musician-59933 · Answer

< bitter alarm 89642> Good question and you ve already found the current answer in that docs link I recognize that that s hardly a satisfying solution though and as it happens I m working on a much better native solution to this problem as we speak This will let you run `meltano extractor tap postgres some source inherit from=tap postgres` to create multiple instances inheriting from the same base plugin If you also define a plugin with `name tap postgres` in your `meltano yml` like the one you have today all plugins with `inherit from tap postgres` will even inherit their parent s configuration so you can store shared config in once place slightly smiling face

bitter-alarm-89642 · Answer

thanks for the quick response < ripe musician 59933> I must say I am liking Meltano so far quite much for its modular design ok hand I tried three to four other projects Was a little surprised on realizing that there s no native support for this already This seems like a day0 need Anyway this^ is the natural solution to solve for this problem How close are we to get it out Although I can get myself moving by using pipeline specific configuration but this^ looks much cleaner If it s gonna see the light of the day in the near future I shall plan accordingly If there are independent bits of it where I could contribute to fast track it do let know raised hands

ripe-musician-59933 · Answer

< bitter alarm 89642> The feature works perfectly in that branch I m just putting the final touches to the tests and docs hoping to release this on Monday slightly smiling face

ripe-musician-59933 · Answer

If you re brave you could install meltano directly from the branch or the dynamically built Docker image

ripe-musician-59933 · Answer

gt I must say I am liking Meltano so far quite much for its modular design ok hand I tried three to four other projects That s great to hear Looking forward to more feedback and contributions as you get to play around with it more slightly smiling face

ripe-musician-59933 · Answer

< bitter alarm 89642> took me a little longer to get over the finish line than expected but I promise it was worth the wait