Hello, Anyone already managed to: 1) use inline stream maps to alias stream? For example:

plugins:
  extractors:
  - name: tap-salesforce
    variant: meltanolabs
    pip_url: git+<https://github.com/meltanolabs/tap-salesforce.git@v1.5.1>
    select:
    - OpportunityHistory.*
    config:
      stream_maps:
        OpportunityHistory:
          __alias__: opportunity_history

2) To declare a plugin inheriting from another plugin? For example:

plugins:
  loaders:
  - name: target-bigquery
    variant: z3z1ma
    pip_url: git+<https://github.com/z3z1ma/target-bigquery.git@0.6.8>
    dataset: public
      project: ${GOOGLE_CLOUD_PROJECT}
      location: ${GOOGLE_CLOUD_LOCATION}
  - name: target-bigquery-github
    inherit_from: target-bigquery
    config:
      dataset: github

Hi @quentin_gaborit!

1) use inline stream maps to alias stream?

That variant of tap-salesforce doesn't support stream maps: https://hub.meltano.com/extractors/tap-salesforce/#capabilities. We're working on a revamped version of that connector based on the SDK that does support them, so stay tuned 🙂 .

2) To declare a plugin inheriting from another plugin?

I'm not sure what problem you're having but the parent plugin seems to incorrectly define config outside of the

config

key:

plugins:
  loaders:
  - name: target-bigquery
    variant: z3z1ma
    pip_url: git+<https://github.com/z3z1ma/target-bigquery.git@0.6.8>
    config:
      dataset: public
      project: ${GOOGLE_CLOUD_PROJECT}
      location: ${GOOGLE_CLOUD_LOCATION}

Damn @edgar_ramirez_mondragon you're right about that tap capabilities, my bad.

1) use inline stream maps to alias stream?

That variant of tap-salesforce doesn't support stream maps: https://hub.meltano.com/extractors/tap-salesforce/#capabilities. We're working on a revamped version of that connector based on the SDK that does support them, so stay tuned 🙂 .

How about using it in the bigquery target then? With something like:

plugins:
  loaders:
  - name: target-bigquery-salesforce
    inherit_from: target-bigquery
    variant: z3z1ma
    pip_url: git+<https://github.com/z3z1ma/target-bigquery.git@0.6.8>
    config:
      dataset: salesforce
      stream_maps:
        OpportunityHistory:
          __alias__: opportunity_history

That seems to work because it actually starts creating the tables with the proper name. However before the run completes I get the following error:

A record for stream 'OpportunityHistory' was encountered before a corresponding schema

2) To declare a plugin inheriting from another plugin?

I'm not sure what problem you're having but the parent plugin seems to incorrectly define config outside of the

config

key:

Yeah sorry I messed up the copy/paste, the configuration is actually correct. I expected that inheritance would allow to overwrite some parameters without having to recreate a plugin venv. To be specific right now although I only use bigquery, I figured my only way to write my tap outputs to different datasets, I need to duplicate the target-bigquery in as many version as I have taps.

For 1) you can use https://github.com/MeltanoLabs/meltano-map-transform to do stream map operations (like aliasing) with non-SDK taps.

Hey @Matt Menzenski thanks for the link, it seems to do the trick for the stream renaming. Have you used mappers extensively yourself? Are you aware of any elegant way to perform generic mapping such as

CamelCase

to

snake_case

? Ideally for both streams and stream properties?

I found this issue https://github.com/MeltanoLabs/meltano-map-transform/issues/148 that links to https://github.com/meltano/sdk/issues/1350. Theoretically you can write a custom mapper to do this also