Good Evening All, I am currently working on piping some data in Snowflake using

target-snowflake --variant meltanolabs

every source DB and table map to the same schema in Snowflake regardless of source schema: •

source_table

--> Fully qualified source:

some_db.some_table

•

source_pk

--> PK of the source table:

some_column

•

source_data

--> all source data as a VARIANT (JSON) •

_sdc_*

I can get this to work using

meltano-map-transformer

and configuring every stream:

mappers:
    - name: meltano-map-transformer
      variant: meltano
      pip_url: git+<https://github.com/MeltanoLabs/meltano-map-transform.git>
      mappings:
        - name: source_data
          config:
            stream_maps:
              <stream_name>:
                source_table: '"some_db.some_table"' 
                source_pk: '"some_column"'
                source_data: record
                __else__: __NULL__
                __alias__: src__some_table

The issue is we have to create a map for every stream (There are many). I know I can create a quick script to create these configs however I was looking for a more dynamic way to generate these where I only have one mapping that is applied to every stream where the values for

source_table

source_pk

and

__alias__

is set dynamically. Any assistance or pointer would be greatly appreciated.

Hi @John Archer! So both the

source_table

and

__alias__

are a function of

<stream_name>

? In that case it's not currently supported. The first is probably easy to add support for (I filed #2283), but

__alias__

is more complicated since we're not currently evaluating expressions there, and doing so is non-trivial and I suspect tough to implement without introducing breaking changes.

Thank for this, I have been smashing my head. With that change then a stream_map_config look up would be possible kind of like

config[___stream_name__]

to get the source_key_

Yeah, but you'd still need to hardcode

__alias__

so you'd need to do it for every stream. There's also https://github.com/meltano/sdk/pull/1888, which would allow you to apply a single stream map to any matching streams.

Thank you @Edgar Ramírez (Arch.dev) for the info and support. I will keep an eye out for these changes.