This is more of a generic architectural question, so I'm putting it here. Is my understanding correct that the only significant data stored in the meltano database currently are the jobs, and in particular state information? That in order for a elt run to succeed you actually need the meltano.yml file populated with extractors, loaders, and transformers?

That's a pretty good summary of it in my opinion. We run our pipelines in a "serverless" mode, where the container mounts the latest project repo, installs dependencies, reads the state from the database, runs the

elt

job, then writes the state back to the database. So the only thing that's being passed from job to job (via the database) is the job state information.

OK, so if I have kubernetes spinning up pods for the runs, then the container image has to have all extrators, loaders, dbt plugins, all in that one container? And I'd need to render a .env file with appropriate secrets (of course not in the container image) for every job run? So changing any parameters for any single run requires a new container image that all other jobs would use also? And, if we do render secrets into .env with say Vault Agent (which I actually have working), that means that every run has access to all secrets, regardless of what extractor/loader/transformer they are used with? I think I know the answer to all of these questions, just thinking that the current architecture won't scale very well. Can we even change the schedule, since MELTANO_PROJECT_READONLY Is set to 1 in the Dockerfile? So even changing the schedule would require a new image?

so we use Argo Workflows to execute all of our jobs on Kubernetes. The advantage with that is that the container with the extractor/etc. installations is built at runtime, rather than built and published to a repository in CI/CD. The parameters themselves are quite flexible, so for example, you could deploy one pod with

TARGET_SNOWFLAKE_DATABASE=foo

and another with

TARGET_SNOWFLAKE_DATABASE=bar

and your job would write to different databases. The

.env

file is more of a convenience for local development -- in prod you could just run it with the environment variables set at runtime. The schedule itself would be more up to how your jobs are going to get orchestrated. That might be through Airflow, for example, or in our case, Argo Workflows.

Hmm. So I re-read your post. Interesting that you "mount the latest project repo," so I assume no new container for changes But it's still a single project repo, correct? I'd prefer to have each job separate, so different teams can work independently and not risk disrupting other runs by making some change that effects them.

There's a gist in here of our configuration: https://meltano.slack.com/archives/CMN8HELB0/p1628034231012200

Thanks. That's what I started doing as the first try running this, naming the extractors and loaders, etc, with the client/tenant name. I'll check it out and work on breaking things out later. Appreciate it! 👍