Hello my good Meltano community! I'm trying to set multiple schema definitions via environment variables. How would I structure the

<EXTRACTOR>__SCHEMA

variable to include multiple different streams with different column definitions? I'm looking specifically at this documentation.

hello!

Lol hit enter by mistake 😆

export <EXTRACTOR>__SCHEMA_<ENTITY>_<ATTRIBUTE>_<KEY>=<value>

seems to be the answer but, full disclosure, I’ve never done this. do you have the different stream definitions?

It said this right above that:

# Once schema descriptions have been set in `meltano.yml`, environment variables can be used
# to override specific nested properties:

oh that’s in the code block. hmm, I don’t know the answer then. My gut would say that you could just set the environment variable and that would work (that’s what would make sense to me) but that text there seems to indicate otherwise. @douwe_maan can you chime in?

Hi, @ricky_renner - just to confirm, did you want to set up two instances of the extractor, with one inheriting from the other, or did you want a single instance that can target two different schemas depending on the env vars when run?

@ricky_renner So you want to set the entire schema for a tap and its multiple streams using a single env var? You're on the right track with

TAP_FOO__SCHEMA

, which you'll want to set it to a JSON-encoded dictionary with a top-level key for each stream. If you want to use a separate env var for each stream, you'll first have to add a key for each stream to

meltano.yml

, which can look like this:

schema:
  stream_one: {}
  stream_two: {}

You can then use

TAP_FOO__SCHEMA_STREAM_ONE='{"<attribute>": {"<key>": "<value>"}'