https://meltano.com/ logo
#announcements
Title
# announcements
d

delightful-soccer-65673

04/14/2021, 6:26 AM
Is there a way to do something like this with environment variables? (this doesn't work)
Copy code
plugins:
  extractors:
  - name: tap-postgres
    variant: transferwise
    pip_url: pipelinewise-tap-postgres
    env:
       PGSSLMODE: verify-ca
       PGSSLKEY: '/home/meltano/ssl/production/product/psql.key'
       PGSSLROOTCERT: '/home/meltano/ssl/production/product/ca.crt'
       PGSSLCERT: '/home/meltano/ssl/production/product/psql.crt'
    config:
      dbname: xx
      host: xx
      port: xx
      ssl: true
      user: xx
      default_replication_method: FULL_TABLE
  loaders:
  - name: target-postgres
    variant: transferwise
    pip_url: pipelinewise-target-postgres
    env:
       PGSSLMODE: verify-ca
       PGSSLKEY: '/home/meltano/ssl/production/analytics/psql.key'
       PGSSLROOTCERT: '/home/meltano/ssl/production/analytics/ca.crt'
       PGSSLCERT: '/home/meltano/ssl/production/analytics/psql.crt'
    config:
      logging_level: DEBUG
      user: xx
      host: xxx
      dbname: xxx
      default_target_schema: xxx
      add_metadata_columns: true
      parallelism: 16
Basically the postgres plugins don't have a config for SSL certs, when the tap and target are the same I can just use the .env file and specify the certs globally but if the tap and target are on different servers, the SSL certs are different
r

ripe-musician-59933

04/14/2021, 5:11 PM
@delightful-soccer-65673 Good question! If we know a plugin supports some env vars, I think we should define settings for them in
discovery.yml
(https://gitlab.com/meltano/meltano/blob/master/src/meltano/core/bundle/discovery.yml#L866) like so:
Copy code
settings:
- name: ssl_mode
  env: PGSSLMODE
- name: ssl_key
  env: PGSSLKEY
# etc
Any values set for those settings using
meltano config
or some other mechanism will be included in the tap/target's 
config.json
, where they would presumably be ignored, but they'd also be set in the plugin invocation env using the name specified by 
env
. And like any setting, they can be set in 
meltano.yml
.env
, the system database, etc, instead of having to be hard-coded under your suggested
env
property. Would you like to create an issue to add those setting definitions to
discovery.yml
for this tap and target?
d

delightful-soccer-65673

04/14/2021, 7:50 PM
@ripe-musician-59933 the interesting thing with this is that the plugin doesn't officially support these env vars (they aren't mentioned in the code). I believe the underlying postgres libraries pick them up as default (PGSSLKEY etc)
When I get time to add the documentation i'm going to create a PR for my changes 🙂
r

ripe-musician-59933

04/14/2021, 8:10 PM
Ah yeah, makes sense that it's the underlying library. From the user's perspective they're still settings on the tap/target though 🙂