Do you happen to have a link to somewhere that describes how the config object is passed around in meltano / singer @douwe_maan? I am running into an issue with transformations for my current fix of the snowflake target and I would like to better understand how the communication flows between taps, targets, and dbt.

@nevin_morgan Which config object are you referring to specifically? Individual plugins are passed the config you can see in

meltano config <plugin>

using both

config.json

(in the case of taps and targets) and env vars (matching the names listed under

meltano config <plugin> list

), and within a

meltano elt

pipeline, the loader and the transformer have access to extractor and loader config through these special pipeline environment variables: https://meltano.com/docs/command-line-interface.html#pipeline-environment-variables

I think I am looking for the config.json and how it is generated then. That is definitely a helpful breakdown and I think I need to spend some time reading the meltano code to really get what is going on internally

To invoke a plugin, we call

PluginInvoker.prepare()

: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/plugin_invoker.py#L93 followed by

PluginInvoker.invoke(*args)

: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/plugin_invoker.py#137

prepare

sets

self.plugin_config_processed

to the exact same JSON object you see when you run

meltano config <plugin>

, after which it calls the

configure

hook on the plugin, and then the

ConfigService.configure()

: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/plugin/config_service.py Singer taps and targets implement the

before_configure

hook to dump

invoker.plugin_config_processed

into a JSON file at

.meltano/<type>/<name>/tap.config.json

: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/plugin/singer/base.py#L26

ConfigService.configure()

then copies config files from

.meltano/<type>/<name

to `.meltano/run/<plugin>`: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/plugin/config_service.py#L44 Finally,

PluginInvoker.invoke()

calls the plugin's

exec_args()

method to determine the args to call the executable with: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/plugin_invoker.py#L119, which in the case of Singer taps and targets adds the

--config

flag and the config file path: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/plugin/singer/tap.py#L84