Hi, I am exploring

meltano

for one of our products. The product is a multi-tenant SaaS application. Each tenant will have one or more data sources from which the data needs to be extracted on a scheduled basis and loaded in to a common destination. The way I am thinking is there will be predefined set of extractors (say oracle, ms-sql & csv) and loaders(parquet) which will be configured. Since its multi-tentant application I cannot store upfront the different configurations or environment variables. There is a separate application where the users configure the data-sources by providing the credentials. I am looking at storing connection details and other config like state for the source database systems for each tenant externally maybe in a keyvault/database. Once this is done I am hoping there would be a way to invoke the etl pipeline by passing these configurations like

meltano etl tap-postgres <config> target-json <config>

Is there any recommendation to handle these use-cases ?

I’ve used chamber in the past together to do something like

chamber exec <service> -- meltano run ...

exec

populates the environment with the secrets from the specified services and executes the given command. Secret keys are converted to upper case (for example a secret with key

secret_key

will become

SECRET_KEY

).

So you could create a separate chamber service for each tenant, with secrets like

tap_mssql_password

which will be converted to the right env vars by

exec

.

I also worked on a similar multi tenant application where we collected/stored the credentials as part of the application. Then we retrieved them and prepped them the way the tap/targets expected and passed them in as env vars when we called meltano with subprocess. That worked pretty well for us

Hey @sharath_chandra DM me if you want to have a chat about our experience doing this. Happy to share.

@edgar_ramirez_mondragon thanks. I wasn't aware of

chamber

However

chamber

looks like has a dependency on

aws

keyvault. In our case we may deploy on

azure

or

on-prem

. Let me see if there is any cross platform service similar to

chamber

Looks like you've got it, this is similar to what we do.