I have to repeat a lot of plugin configuration for each envi Meltano #troubleshooting

I have to repeat a lot of plugin configuration for...

jan_soubusta

03/31/2023, 2:35 PM

I have to repeat a lot of plugin configuration for each environment even though most of properties are the same in all environments. Example:

Copy code

- name: dev_local
  config:
    plugins:
      extractors:
      - name: tap-salesforce
        config:
          api_type: BULK
          is_sandbox: true
          select_fields_by_default: false
          start_date: '2023-01-01T00:00:00Z'
          streams_to_discover:
          - Account
          .....

The only property which differs in environemnts is the

start_date

. Is there a way how to specify the same properties in the main config and

start_date

per environment? If I am not mistaken, if I specify config both in main plugin config and in each environment, the environment config fully replaces the main one and it obviously does not work then....

Andy Carter

03/31/2023, 2:41 PM

https://docs.meltano.com/guide/configuration#specifying-environment-variables Could you just specify the different variables under the

env:

level and have top-level config the same?

jan_soubusta

03/31/2023, 2:43 PM

Interesting. If I specify ENV_VAR in each environment, will it be expanded when used in the main plugin config?

jan_soubusta

03/31/2023, 2:44 PM

I'll try it

jan_soubusta

03/31/2023, 3:05 PM

Looks like it works! Man, you made my day(and the following weekend) 😉 Design:

Copy code

environments:
- name: dev
  env:
    START_DATE: "2023-01-01T00:00:00Z"
    SALESFORCE_SANDBOX: "true"
- name: prod
  env:
    START_DATE: "2021-01-01T00:00:00Z"
    SALESFORCE_SANDBOX: "false"
plugins:
  extractors:
  - name: tap-salesforce
    variant: meltanolabs
    pip_url: git+<https://github.com/jaceksan/tap-salesforce.git>
    config:
      api_type: BULK
      is_sandbox: ${SALESFORCE_SANDBOX}
      select_fields_by_default: false
      start_date: ${START_DATE}
      streams_to_discover:
        - Account

pat_nadolny

03/31/2023, 3:11 PM

@jan_soubusta you should be able to do what you want without using env. For example I set a default start date at the top level of my config along with other configs, then override just start date in my CICD environment config

Andy Carter

03/31/2023, 3:11 PM

Yes, I don't think you need env variables for this @jan_soubusta

jan_soubusta

03/31/2023, 3:13 PM

I am pretty sure that it did not work exactly in the case of tap-salesforce. But I may be wrong, will give it a try once more 😉

pat_nadolny

03/31/2023, 3:25 PM

@jan_soubusta if its not working as expected then I'd call it a bug so feel free to open an issue with what youre seeing

2 Views

Open in Slack

Previous Next