Is there a full spec for the meltano.yml file anyw...
# getting-started
f
Is there a full spec for the meltano.yml file anywhere? I see the web page docs, which go into a lot of the configuration, but I'm looking for an actual spec (like a json schema or similar). What I'm trying to figure out if I can do is configure a settings entry for an array where some elements are not sensitive, but some are. I'm using this as a reference: https://gitlab.com/DouweM/tap-investing/-/blob/master/tap_investing/tap.py#L34 . In my use-case some of the properties would be an API key. So the tap would connect to a service that was multi-tenant and pull data for all tenants, each of which has a separate API key (and other information). Is this an anti-pattern, and should I be creating multiple ELT pipelines, one for each customer? They would all end up in the same data warehouse/lake tables, so I'm thinking they should all be processed in the same pipeline. If we have 400 tenants, we wouldn't want to create 400 separate pipelines, I wouldn't think...
I looked at the source, and it doesn't look like you can define multi-tier settings of arbitrary depth in the meltano.yml file.
e
Hi @fred_reimer, • Regarding a json schema spec for
meltano.yml
, there's #2211. An MR was started by a community member but later abandoned. It's still open to contribution simple smile. • For an array of settings where at least one element is sensitive, Meltano doesn't provide a way to tell them apart, so it'd all be an array of plain strings. • On the topic of multi-tenant pipelines, @ken_payne does a good job of explaining a good pattern supported by Meltano: https://meltano.slack.com/archives/C01E3MUUB3J/p1633537536022500. Let me know if that's helpful or if you have any questions