https://linen.dev logo
Join Slack
Powered by
Hi everyone! Very excited to be part of this commu...
# troubleshooting
m
Hi everyone! Very excited to be part of this community! I am a developer myself, and have the opportunity to build a new integration pipeline for my company. Meltano is very interesting and have decided to use it for our use case. However, there is something I am really struggling with, which is something as simple as creating custom environment variables that for development can be stored in the 
.env
and on production they are injected via docker. How can be done? I have only seen environment variables for plugins. An example 
meltano.yml
for clarification of what I want to do:
Copy code
version: 1
default_environment: dev
project_id: <project_id>
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-rest-api-msdk
    variant: widen
    pip_url: tap-rest-api-msdk
    config:
      api_url: https://******
      next_page_token_path: $.data.next
      streams:
      - name: dev_partner
        headers:
          Authorization: Bearer $TOKEN
          X-Integration-Id: $INTEGRATION_ID
        params:
          include_deleted: 'true'
          page_size: 250
        path: /v1/data
        primary_keys:
        - id
        records_path: $.data.results[*]
        num_inference_records: 250
    select:
    - '*.*'
  loaders:
  - name: target-postgres
    variant: transferwise
    pip_url: pipelinewise-target-postgres
And the `.env`:
Copy code
TOKEN=12345
INTEGRATION_ID=123kljh123kj1h23k1j23h1k2j3h
Thank you!
a
Did you see the docs here? https://docs.meltano.com/guide/configuration#environment-variable-expansion You should be able to set an env variable under the environments: key and that should make available to all plugins under that environment I think
Copy code
environments:
  - name: prod
    env:
      # Level 3: environment-level `env:`
      # Inherits from top-level `env:`
      LEVEL_NUM: "3"              #   '3'
      STACKED: "${STACKED}3"      # '123'
m
That would allow me to have env vars for each environment. Is there a way to have them for all environments at once?
I see, like this:
Copy code
env:
  # Level 2: top-level `env:`
  # Inherits from terminal context
  LEVEL_NUM: "2"                  #  '2'
  STACKED: "${STACKED}2"          # '12'
plugins:
  extractors:
    - name: tap-foobar
      env:
        # Level 4: plugin-level `env:`
        # Inherits from a environment-level `env:` if an environment is active
        # Inherits directly from top-level `env:` if no environment is active
        LEVEL_NUM: "4"            #    '4'
        STACKED: "${STACKED}4"    # '1234'
environments:
  - name: prod
    env:
      # Level 3: environment-level `env:`
      # Inherits from top-level `env:`
      LEVEL_NUM: "3"              #   '3'
      STACKED: "${STACKED}3"      # '123'
    config:
      plugins:
        extractors:
          - name: tap-foobar
            env:
              # Level 5: environment-level plugin `env:`
              # Inherits from (global) plugin-level `env:`
              LEVEL_NUM: "5"          #     '5'
              STACKED: "${STACKED}5"  # '12345'
u
@manuel_garcia this issue might be relevant https://github.com/meltano/meltano/issues/3171. Its possible your env vars are not being expanded because of that. If you run 
meltano config tap-x
it will print out the final config file thats passed to the extractor so you can see what is being expanded or not
m
I see, that is exactly what I am facing. Do you reckon there is a better way to inject secrets?
u
I know you can put that whole 
streams
config into an environment variable like 
TAP_REST_API_MSDK_STREAMS="{\"blahblah\"}"
. Again I'd test that out using the config command to make sure it prints properly
u
Copy code
TAP_REST_API_MSDK_STREAMS='[ { "name": "dev_partner", "headers": { "Authorization": "Bearer $TOKEN", "X-Integration-Id": "$INTEGRATION_ID" }, "params": { "include_deleted": "true", "page_size": 250 },
 "path": "/v1/data", "primary_keys": [ "id" ], "records_path": "$.data.results[*]", "num_inference_records": 250 } ]' meltano config tap-rest-api-msdk
m
I see, I would prefer to inject just single secrets and let Meltano resolve them when interpreting the yml file, however, this would work. Let me try that and get back to you. Thank you for the assistance!
u
Yeah 100% - this is a work around that I use also until that issue is resolved
u
Also if you wouldnt mind adding a quick comment to https://github.com/meltano/meltano/issues/3171 to mention that you ran into this issue and any other thoughts you have. It will become higher priority to get fixed if we hear more people are running into it, and I also want it fixed 😉
m
That is a good shout. Will do!
2 Views
Open in Slack
PreviousNext
Powered by