EDIT Fixed it I think I m having issues trying to use `inclu Meltano #troubleshooting

EDIT: Fixed it I think! I'm having issues trying ...

Andy Carter

05/31/2023, 11:48 AM

EDIT: Fixed it I think! I'm having issues trying to use

include_paths

to have a central meltano.yml and then separate yml files for dev and prod config, following the example in meltano_squared. When I run

meltano compile

I can't see the config being overwritten correctly if I use a 'environment' in the top of the included yml. Any suggestions as to what I am doing wrong? Files below. If I remove the

environments

part of the included yaml, it works as intended (but obviously I can't change the config depending on my chosen env to run, which was the point in the first place!)

Andy Carter

05/31/2023, 11:49 AM

My meltano.yml

Copy code

version: 1
include_paths:
  - ./meltano-yml/dynamics*.yml
default_environment: dev
project_id: 4**
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-spreadsheets-anywhere
    variant: ets
    pip_url: git+<https://github.com/ets/tap-spreadsheets-anywhere.git>
    config:
        tables: []

dynamics-prod.yml (this doesn't work)

Copy code

environments:
- name: prod
  config:
    plugins:
      extractors:
      - name: tap-spreadsheets-anywhere
        config:
          tables:
          - path: azure://***
            name: dynamics__account
            pattern: ^account\/[0-9]{4}-[0-9]{2}.csv$
            start_date: '2017-05-01T00:00:00Z'
            key_properties:
            - id
            format: csv
            prefer_schema_as_string: true
            field_names:
            - Id

dynamics-prod.yml (this DOES work)

Copy code

plugins:
      extractors:
      - name: tap-spreadsheets-anywhere
        config:
          tables:
          - path: azure://***
            name: dynamics__account
            pattern: ^account\/[0-9]{4}-[0-9]{2}.csv$
            start_date: '2017-05-01T00:00:00Z'
            key_properties:
            - id
            format: csv
            prefer_schema_as_string: true
            field_names:
            - Id

Andy Carter

05/31/2023, 11:56 AM

Solution was to remove the

environments

key in the top meltano.yml

Copy code

version: 1
include_paths:
  - ./meltano-yml/dynamics*.yml
default_environment: dev
project_id: 4**
plugins:
  extractors:
  - name: tap-spreadsheets-anywhere
    variant: ets
    pip_url: git+<https://github.com/ets/tap-spreadsheets-anywhere.git>
    config:
        tables: []

Open in Slack

Previous Next