Is there a way to pass configuration into the `meltano elt` Meltano #troubleshooting

Is there a way to pass configuration into the `mel...

jam

01/18/2023, 5:39 PM

Is there a way to pass configuration into the

meltano elt

command (something similar to

--config=ENV

)? I have a

meltano.yml

that uses

password

settings. Those settings are not being read from the

.env

db

causing a

singer_sdk.exceptions.ConfigValidationError

edgar_ramirez_mondragon

01/18/2023, 6:03 PM

Hi @jam! The names of the settings may be incorrect in

.env

. Can you check that they’re in the form

<PLUGIN_NAME>_<SETTING_NAME>

jam

01/18/2023, 6:22 PM

Yes, that is the format that I'm using. I think it's important to note that I'm doing this via the

BashOperator

in Airflow. I'm passing in the environment variables via the

env

argument. This worked when I did

meltano invoke <PLUGIN_NAME> --config=ENV

,but now that I'm doing

meltano elt <PLUGIN_EXTRACTOR> <PLUGIN_LOADER>

it fails. the env variables are named like

<PLUGIN_EXTRACTOR>_<SETTING_NAME>

jam

01/18/2023, 6:22 PM

I've tested with a

.env

file and I'm seeing the same behavior.

edgar_ramirez_mondragon

01/18/2023, 6:23 PM

Can you share your

meltano.yml

jam

01/18/2023, 6:26 PM

Copy code

version: 1
default_environment: dev
plugins:
  extractors:
  - name: tap-x
    namespace: tap_x
    pip_url: xxxxxxx
    capabilities:
    - state
    - catalog
    - discover
  loaders:
  - name: target-duckdb
    namespace: target_duckdb
    pip_url: xxxxxxx
  orchestrators:
  - name: airflow
    variant: apache
    pip_url: apache-airflow==2.4.3 --constraint <https://raw.githubusercontent.com/apache/airflow/constraints-2.4.3/constraints-${MELTANO__PYTHON_VERSION}.txt> apache-airflow-providers-amazon
  files:
  - name: files-airflow
    variant: meltano
    pip_url: git+<https://github.com/meltano/files-airflow.git>
environments:
- name: dev
  config:
    plugins:
      extractors:
      - name: tap-x
        config:
          company: xxxxx
          user: xxxxx
          client: xxxx
          base_url: xxxxx
          api_namespace: xxxx
          headers: xxxxx
          batch_config:
            encoding:
              format: parquet
              compression: gzip
            storage:
              root: xxxxx
              prefix: test-batch-
        settings:
        - name: api_key
          kind: password
        - name: api_password
          kind: password
      loaders:
      - name: target-duckdb
        config:
          sqlalchemy_url: duckdb:///./test_db.duckdb
          default_target_schema: xxxxx
          batch_config:
            encoding:
              format: parquet
              compression: gzip
            storage:
              root: xxxxxx
              prefix: test-batch-
      orchestrators:
      - name: airflow
        config:
          core:
            dags_folder: /workspaces/internal-data-warehouse/airflow/dags
            dags_are_paused_at_creation: true
          webserver:
            web_server_port: '58080'
- name: staging
- name: prod

edgar_ramirez_mondragon

01/18/2023, 6:43 PM

Ok, so custom plugins (those with explicit

namespace

) have to declare `settings`: https://docs.meltano.com/reference/plugin-definition-syntax#settings So, unfortunately you have to re-declare those settings (https://github.com/meltano/meltano/issues/7156)

jam

01/18/2023, 7:08 PM

Thanks, @edgar_ramirez_mondragon! That did the trick!

Open in Slack

Previous Next