Stupid Friday question: I use 1 meltano.yml for both plugin installations during the Docker build and when I execute

meltano run

in the pipeline. I need to split it - when I change e.g. plugin select list of entities, I do not want to trigger the docker build - it should be triggered only if versions of plugins are changed or if a new plugin is added. I tried to create 2 meltano.yml files: • For docker build - Just the list of plugins with pip URLs, no config, no select • For meltano run - complete config of all plugins and environments, without pip URLs. Includes the first one It does not work, at least not for tap-salesforce plugin - the config is not kept correctly after the merge of two configs, default setup is used. How to solve this use case properly?

hey @jan_soubusta. AFAICT that should work, i.e. having an included file at a later docker stage and thus does not trigger a full rebuild of the image. Can you share your yaml contents?

File for docker build - `meltano-build.cfg`:

plugins:
  extractors:
    - name: tap-salesforce
      variant: meltanolabs
      pip_url: git+<https://github.com/meltanolabs/tap-salesforce.git>

File for meltano run - `meltano.yml`:

plugins:
  extractors:
    - name: tap-salesforce
      select:
        - Account.Id
        - Account.Name
        - Account.Type
        - Account.Segment__c
        - Account.Tier__c
        - Account.OwnerId
include_paths:
  - "./meltano-build.cfg"

@jan_soubusta this should work IMHO. We're doing something similar in our installation, maybe you can take a look at that: https://github.com/meltano/squared/blob/main/data/meltano.yml and @pat_nadolny (CC, am I missing something here?).

Hm, this is different, IMO. You include disjoint parts (e.g. environments, loaders, ...). I need to include the same part (e.g. configuration of tap-salesforce) from two files, one containing only variant and pip_url and second containing configuration options (start_date, ..., select). My expectation is that if they contain different sub-keys (e.g. pip_url and config/select), they should be merged into one full configuration). If they contain the same sub-key, the second included should override the previous one.

I think I've set mine up the way you described and its working as expected for me on meltano

2.16.1

Interesting. Will try it once again to be sure that there is an issue.