I know that if you want multiple configurations for the same tap, you can use plugin inheritance to specify multiple configurations, but is there any way to link a unique tap config to a specific pipeline/job name? In the system database I believe the state for each pipeline is stored separately. It would be pretty cool if we can do

meltano config <plugin> --job_id=gitlab_to_postgres set

to set plugin settings specifically for that pipeline. Maybe it already exists? 🤷‍♂️

Hi, @johann_du_toit. We just added support for environments and each environment can override it's own config per plugin - very similar to what you describe but at the environment level, rather than the job ID level. Do you think this would help for your use case?

Thanks @aaronsteers - not sure why I didn't see that before. I guess you still have to create all your environments beforehand in a config file? The use-case I'm coming from is to be able to create and configure things without modifying the base config file. We run similar pipelines for multiple clients so want to separate them with different job_ids. However we also want to be able to update some of the plugin config for each such as access tokens, different target dbs, etc. Preferably storing as much as possible in the system database, rather than files.

Hey, @johann_du_toit. You can also leverage environment variables and a secret storage tool like chamber (or #2918 in the future) to change tokens, target dbs, etc. at runtime:

GITLAB_API_TOKEN=abc
TARGET_SNOWFLAKE_DBNAME=first
meltano elt tap-gitlab target-snowflake --job_id gitlab_to_postgres1

GITLAB_API_TOKEN=def
TARGET_SNOWFLAKE_DBNAME=second
meltano elt tap-gitlab target-snowflake --job_id gitlab_to_postgres2

Thanks @edgar_ramirez_mondragon - yeah, that's what we were going to go with if we couldn't manipulate the system database directly. 🙂 That ticket seems to be exactly what I was after, especially @pat_nadolny’s suggestion for multi-tenancy. We are a multi-tenant saas, hence these needs.