Hey all, i had a question regarding setting stream maps for all streams selected in a tap. We are externally triggering an Airflow DAG that runs a specific tap and target-snowflake depending on the config passed. Due to how this works, we’d like to alias each stream with the client name as a prefix with the stream name in front. Is there a way to apply stream maps to all streams selected without having to hard code each stream? For instance, is there a way to select all (i.e. *) instead of having the stream hard coded:

version: 1
send_anonymous_usage_stats: false
plugins:
  extractors:
  - name: tap-pluralsight
    namespace: tap_pluralsight
    pip_url: git+<https://github.com/repo/tap>
    settings:
  # Stream maps
    - name: stream_maps.pluralsight_contentcatalog.__alias__ ### want this to apply to all streams without hard coding
      value: ${CLIENT_NAME}_${STREAM_NAME}
    capabilities:
    - state
    - catalog
    - discover
    config:
      api_token: $INTEGRATIONS_TAP_PROVIDER_API_TOKEN

Is there a way to do something like:

# Stream maps
    - name: stream_maps.*.__alias__ 
      value: ${CLIENT_NAME}_${STREAM_NAME}

So that this logic gets to applied to all/any stream that gets passed?

I think I asked a similar question before, and I think the answer was 'probably no': https://meltano.slack.com/archives/CMN8HELB0/p1677852788680639 Some targets might allow this but it is implemented on a plugin-by-plugin basis, not by stream map. Would be delighted to be shown otherwise though 🙂

Yeah, @Andy Carter is right. This isn’t possible stream-maps or the meltano-map-transformer at the moment. You could however create a simple mapper application that adds a prefix to all your streams. Something like https://github.com/edgarrmondragon/singer-playground/blob/6a4179bce36d43121d82f1d8d56aca1a3c9183b0/merge_streams/map.py.

Thanks @Andy Carter and @edgar_ramirez_mondragon! @Andy Carter You mentioned some targets could be implemented with that functionality. Is that something you ended up doing by chance? We are using a forked version of target-snowflake that i’d be willing to add that logic to.

Hi @bryan_rose, I didn't add this functionality to targets, but the target plugin developers added them. It's super useful for organisation within a single db schema. Off the top of my head, some supporting

prefix

• target-mssql • target-bigquery (adswerve variant)