Hello all, I hope everyone has been well. I'm hoping to get some advice on whether Meltano would be suitable for my data integration use case. In my SaaS application, I would like to provide the option to my clients to add their Google Sheets spreadsheets into my web interface. Basically, a data import of sorts. In my backend, I'm then looking to perform some basic SQL transformations using Postgres, and provide data according to the client's frontend configuration. My main question is, given that I am looking to ingest possibly many different spreadsheets from my clients (i.e. many sources) with each a different layout into a database, say, Postgres, would Meltano be suitable? I've successfully gone through a simple Meltano workflow using a

tap-google-sheets

extractor and a

tap-postgres

loader and it worked rather smoothly. My doubts lie mainly in: • Use of the CLI interface (looking at the docs, not too sure how to integrate it into my Rust backend). Also, since each client Google account requires a different

refresh_token

, I'm not too sure on how easy it is to generate the Meltano configuration file after redirecting the users to Google authentication within my app. • Whether there would be difficulty in allowing for an arbitrary number of sources, as created "on-the-fly" by my clients. Naturally, I am looking to extend my application to allow clients to pull in data from more than just the Google Sheets API, hence I am looking to Meltano as a possible solution to make data integration easier.

@aaronsteers thoughts on this? My first thought is you could use plugin inheritance to simplify some of the config generation and then just have the refresh_token look to an environment variable that you set for running.

@tedmund_ho If I understand correctly your use case is similar to this https://meltano.slack.com/archives/C01E3MUUB3J/p1633539400023100?thread_ts=1633537536.022500&cid=C01E3MUUB3J. It sounds like its becoming a common pattern. I've also personally done something like that where my application manages user credentials and then just injects them as environment variables into meltano at runtime. So basically one generic loader config in your meltano.yml

tap-google-sheets

can support many different user specific instantiations on the fly.

@tedmund_ho - just to chime in on agreement with the above, you can definitely use Meltano for this. Depending on number of customers and similarity of pipelines, you can decide to have separate configurations for each customer, for each subgroup of customers, or a single config for each to reuse. When reusing config with separate auth tokens per environment, people will generally store the creds in a secret manager and then inject the correct token at runtime as environment variables.

@aaronsteers @pat_nadolny @taylor Wow, thank you all for the comprehensive responses! I’ll try out the multi-user configuration and see how things go!