hi, good morning, what's the best practice for a m...
# infra-deploymentm
marc_garcia_sastre
10/05/2021, 2:20 PMhi, good morning, what's the best practice for a multi tenant support, would a setup with this https://meltano.com/docs/project.html#inheriting-plugin-definitions be enough? thanks
a
aaronsteers
10/05/2021, 2:24 PMCan you say more about what you mean by multi-tenant? Do you mean some resources in AWS and some in GCP/Azure? Or something else?
aaronsteers
10/05/2021, 2:24 PMcc @ken_payne
m
marc_garcia_sastre
10/05/2021, 2:26 PMWe want to provide integrations to several customers, that means possibly have several extractors which might be the same but for different clients
a
aaronsteers
10/05/2021, 2:48 PMAh yes - that makes sense. Thanks for clarifying. For this, you can certainly use plugin inheritance. If the number of instances you want gets larger than 10-20, you would probably want to lean more on environment variable-based config. In this model, you can have a small number of reference configurations but when invoked, you send different environment variables corresponding to each customer and their data's target landing space.
m
marc_garcia_sastre
10/05/2021, 3:16 PMThanks for clarification, but for what I have tried so far, the only way to have this kind of setup is via yaml configuration, I haven't found a way to do it with bash or via web UI. In our setup we want to let the user decide from certain available options from where to get their sources and with that we import the sources for the user. TL:DR; we want to let the users configure their extractors
marc_garcia_sastre
10/05/2021, 3:19 PMyou would probably want to lean more on environment variable-based configcan you point me to the documentation on this, seems like the perfect setup for us
e
edgar_ramirez_mondragon
10/05/2021, 3:34 PM@marc_garcia_sastre the docs are here: https://meltano.com/docs/configuration.html#environment-variables. Basically if you have a tap
my-tapsome_configMY_TAP_SOME_CONFIGm
marc_garcia_sastre
10/05/2021, 3:34 PMthats perfect, thanks
a
adam_rudd
10/06/2021, 2:03 AMhey re: inheritance - is there a reason why that isn’t the default approach via the UI?
(as in, when I look at the
meltano.ymla
aaronsteers
10/06/2021, 3:17 AM@adam_rudd - Do you mean that editing within
meltano uia
adam_rudd
10/06/2021, 3:19 AMI think it’s updating correctly; its more that any meltano config added uses the plugin entry, so if I wanted to add another tap/target of the same type (eg. S3 csv) i can’t
adam_rudd
10/06/2021, 3:22 AMI’ll pop in a feature request and see if I can do a PR for the change if you folks agree.
use case:
• team has 2 pipelines they want to create via meltano UI
• Target is s3 for both pipelines
• Team creates pipeline 1 with s3 Target
• When the Team goes to create pipeline 2, they cannot use the same s3 Target (its greyed out) because the plugin settings in the yaml has been attributed to pipeline 1
fix:
• team has 2 pipelines they want to create via meltano UI
• Target is s3 for both pipelines
• Team creates pipeline 1 with s3 Target. A new inherited entry is created in the
meltano.ymlmeltano.ymls
sumit_rai
10/06/2021, 2:48 PMI think you'll have problem in security in the future. All the passwords of the extractos are stored in .env file, and that can be mess.
k
ken_payne
10/06/2021, 3:28 PMCompletely agree @sumit_rai - managing secrets in plain
.enva
aaronsteers
10/06/2021, 5:06 PM@marc_garcia_sastre (et al) - This was a hot topic in #C01QS0RV78D today! If you missed it, you can check the recording and scan to maybe 20 or 30 minutes, until we start talking about "Environments" 🙂
cc @keegan_mccallum