Our

select:

list in

meltano.yml

is getting pretty long and making our yml a bit large…is there a way to link the select yml portion to an external page/yml? Something like :

select: extract/tables/select.yml?

or something along those lines? This would also be helpful when wanting to pass different

select

lists depending on what environment we are running

https://docs.meltano.com/concepts/project#multiple-yaml-files yes

@visch thanks! so if i add the included path for a select yml (

select_production.yml

), how would i make sure

select:

in

meltano.yml

use that file?

include_paths:
  - './select_production.yml'
select: ?

If you want exactly that maybe look at YAML anchors

Ok cool ill look into it, thanks!

I don't think we'd be able to pull in a yaml anchor from another page as of now, unfortunately. I think each page is evaluate separately.

yeah i just tested and looks like it can't be done

But - you could move just that tap into its own file. Perhaps not ideal, but could help to keep the rest of the file easier to read

thanks @aaronsteers

You could even just define each extractor in its own file as a pattern:

include_paths:
  - './extractors/*.yml'

Definitely. We finally wrapped up fastsync as well so i will pop in and update what u sent before

Sorry I didn't read closer! 😄 glad you have a way forward

hey @aaronsteers i created a dir

extractors

with the new yml,

prod_test.yml

, which i have added to the included paths. Here is how i have it setup in

meltano.yml

now:

version: 1
send_anonymous_usage_stats: false
project_id: 0ebb5876-102f-4a42-a66a-f6b317821032
include_paths:
  - extractors/prod_test.yml
plugins:
  extractors:
  - name: pipelinewise-tap-mssql

In the prod_test.yml, i added the pipelinewise-tap-mssql extractor yml, but i am most likley doing this incorrectly. Was trying to find some more documentation to help, but would you mind taking a quick look and seeing where/how this needs to be assigned better? This is what exists in `prod_test.yml`:

extractors:
  - name: pipelinewise-tap-mssql
    namespace: pipelinewise_tap_mssql
    pip_url: git+<https://github.com/degreed-data-engineering/pipelinewise-tap-mssql>
    executable: tap-mssql
    capabilities:
    - catalog
    - discover
    - state
    settings:
    - name: string
    - name: integer
    - name: boolean
    - name: date-time
    config:
...
...
... (rest of the config)

If i try to install the extractor after its setup like this, i get

Extractor 'pipelinewise-tap-mssql' is not known to Meltano

(i am installing it with

meltano install extractor pipelinewise-tap-mssql

which typically works when not doing it this way.