I am getting error ```Plugin with path environments prod alr Meltano #troubleshooting

I am getting error ```Plugin with path "environmen...

atif_imam

09/06/2022, 5:46 AM

I am getting error

Copy code

Plugin with path "environments:prod" already added in file /run/media/atif.imam/data/pycharm_projects/data-integration/source_config/file_1.yml.

meltano-version: 2.5.0

while trying to use environments. My

meltano.yml

looks like this

Copy code

environments:
- name: dev
- name: uat
- name: prod
send_anonymous_usage_stats: true
project_id: 488164fd-cb5b-4fe1
include_paths:
  - ./source_config/file_1.yml
  - ./source_config/file_2.yml

plugins:
  extractors:
    - name: tap-mysql
      variant: transferwise
      pip_url: pipelinewise-tap-mysql
      config:
        ..
        ..
  loaders:
    - name: target-snowflake
      variant: transferwise
      pip_url: pipelinewise-target-snowflake
      config:
        add_metadata_columns: true
        hard_delete: true
        ..
        ..

file_1.yml

and

file_2.yml

in thread.

atif_imam

09/06/2022, 5:50 AM

file_1.yml

Copy code

environments:
  - name: prod
    config:
      plugins:
        extractors:
          - name: tap-mysql-db1
              
        loaders:
          - name: target-snowflake-db1
            config:
              s3_key_prefix: snowflake/db1/
  - name: uat
    config:
      plugins:
        extractors:
          - name: tap-mysql-db1
              
        loaders:
          - name: target-snowflake-db1
            config:

plugins:
  extractors:
    - name: tap-mysql-db1
      inherit_from: tap-mysql
  loaders:
    - name: target-snowflake-db1
      inherit_from: target-snowflake
      config:
        default_target_schema: db1

atif_imam

09/06/2022, 5:51 AM

file_2.yml

Copy code

environments:
    - name: prod
      config:
        plugins:
          extractors:
            - name: tap-mysql-db2
                
          loaders:
            - name: target-snowflake-db2
              config:
                s3_key_prefix: snowflake/db2/
    - name: uat
      config:
        plugins:
          extractors:
            - name: tap-mysql-db2
                
          loaders:
            - name: target-snowflake-db2
              config:
  
  plugins:
    extractors:
      - name: tap-mysql-db2
        inherit_from: tap-mysql
    loaders:
      - name: target-snowflake-db2
        inherit_from: target-snowflake
        config:
          default_target_schema: db2

mayank_sardana

09/06/2022, 6:04 AM

After looking at the code here, looks like it doesn't allow multiple files having

environments:<env_name>

. That seems to be a bug given it is mentioned in docs that

environments

is allowed. Can anyone confirm this?

edgar_ramirez_mondragon

09/06/2022, 3:31 PM

Hi @mayank_sardana! You’re declaring the

prod

environment in both files, so it’s not clear how Meltano should split them back after they’re merged into a single object under the hood (here).

mayank_sardana

09/07/2022, 5:00 AM

Hi @edgar_ramirez_mondragon, thanks for confirming. Here's our use case: We have around 7-8 mysql data sources that we define and maintain in different subfiles. Now, for environments each of the data sources have their own differences when it comes to non-prod vs prod environments. Since meltano doesn't let you parameterise db/schema name under

select

config, we aren't sure how else to follow proper development lifecycle. Is there a better way to handle this? It doesn't seem like an uncommon case to me. Putting them all in a single file will make it very difficult to manage given we have to specify list of tables for each source. Also, it's not intuitive from the docs I mentioned above that we can't have same environments within each subfiles.

edgar_ramirez_mondragon

09/07/2022, 6:07 PM

@mayank_sardana one workaround would be to have one included file per environment (e.g.

prod.yml

and

dev.yml

and remove the environment definitions from the other files. For example:

Copy code

# prod.yml
environments:
- name: prod
  config:
    plugins:
      extractors:
      - name: tap-mysql-db1
      - name: tap-mysql-db2

      loaders:
      - name: target-snowflake-db1
        config:
          s3_key_prefix: snowflake/db1/
      - name: target-snowflake-db2
        config:
          s3_key_prefix: snowflake/db2/

edgar_ramirez_mondragon

09/07/2022, 6:07 PM

(logged https://github.com/meltano/meltano/issues/6727)

4 Views

Open in Slack

Previous Next