Great stuff on the singer spec interpretation. We're a reporting / dashboarding SaaS, and have essentially written our own ELT so far, but would love to adopt or migrate to Singer / Meltano... however the thing holding us back is that everything is so "config file" driven. As an example we have hundreds of Xero clients / connections which we sync every hour - it just seems untenable to have to manage and invoke different config files and executables. We'd love to just pass the config via some API to a service, or invoke taps and targets as libraries - our whole backend is Python already. StitchData obvously does it, so it must be possible with enough code. :-) The Xero tap is a great example, it writes back new OAuth tokens to a file. Currently we store all that in our app's database... the tokens, the state of bookmarks etc. Having to wrangle thousands of config files or worse potentially thousands of custom docker images does not feel "right". I noticed on the Meltano roadmap there's a "SaaS" capability planned for later in the year - does anyone have some more info on the thinking around what that involves? We're happy to throw dev resources at this since we can see the tremendous scaling and maintenance benefits long-term.

Take it for what it’s worth, but we also run a diverse set of integrations and we run meltano with the config’s fully defined by environment variables we pass into the container at runtime. Our

meltano.yml

file is pretty much just what taps we want to be able to run, and the unique config for each run is passed in through env vars at runtime.

chamber is a nice option for these use cases. You store parameters in AWS SSM with paths. If

key

and

url

are

tap_example

config names •

/client1/tap_example_key

•

/client1/tap_example_url

•

/client2/tap_example_key

•

/client2/tap_example_url

And then you can run CLI commands with different "environments":

chamber exec client1 -- meltano elt ...
chamber exec client2 -- meltano elt ...

Hi, @johann_du_toit. I also sent you a DM but want to make sure I loop back on this thread as well. I agree with @edgar_ramirez_mondragon and @paul_blankley here in this thread that env vars are generally a big key in unlocking these in production use cases, especially when running dynamically and with a high cardinality of distinct settings of credential pairs. To your comment regarding multiple docker images, I do often see projects generally having a small number of images being multiplied by a large number of possible configurations - if not fully populated by environment variables, at least seeded by them with distinct tokens. In a past life, I used AWS Secrets Manager and AWS Parameter Store (part of AWS Systems Manager) to dynamically pass env vars to different containers, each running basically the same image but dynamically pulling down different settings from S3 and/or Parameter Store services to get the correct invocation info. You can also create custom bootstrap scripts in the docker image which take care of initializing the environment. I’m curious if these ideas are enough for a path forward, or if you see additional options and/or challenges.

Thanks everyone. I'll check out chamber. We're not on AWS - we use S3 for a few things, but we're trying to stay not too coupled to a specific cloud platform. @paul_blankley so you spin up a container for each sync with the environment variables. Our current sync architecture is very much a "long running daemon process in a container" which takes care of all syncs for everyone. We use Celery to schedule and run the tasks inside our "worker containers". I'm thinking we'll have to change our thinking a bit and make things more atomic rather than so monolithic...

@johann_du_toit Yep, we’re fully AWS, and we use AWS batch to run jobs (that is, docker image of meltano) from triggers, starting from a clean container every time

Was a AWS Batch evangelist at a previous job but I couldn't get my coworkers to sign off. They wanted to jump from where we were to kubernetes, but batch to me always seemed to check all the right boxes without much overhead. I haven't really heard of many people using it so it, so it is nice to hear someone is. I'm also intrigued by what SaaS means for Meltano on their roadmap? Is that just a few key features being implemented and then you can more easily self host a more holistic solution or is it more a business shift where you can buy more features/management of the infrastructure.

@casey The SaaS will be one of the possible ways we'll commercialize Meltano, where it'll be a paid hosted Meltano offering for those who don't want to self-manage, which will compete more directly with the likes of Fivetran