https://linen.dev logo
Join Slack
Powered by
Does anyone have a solution for forcing `env` vars...
# troubleshooting
j
Does anyone have a solution for forcing 
env
vars into a tap config in 
meltano.yml
? I was quite surprised to learn about Meltano’s 
configuration
system which seems to effectively limit the places environment variables can be used. I would need to set a value that is not accessible via the 
tap-rest-api-msdk
. Has anyone solved this with an intermediate script?
v
What are you trying to do exactly? Guessing it's doable but I can point you to the right spot if you give a bit more info 😄
j
I guess is there a way to force a property not supported as a config value initially to be templateable via env vars.
Specific example: I wanna set a param in a stream of the REST API plugin via env var.
v
j
meltano config tap-rest-api-msdk list
outputs
Copy code
Custom, possibly unsupported by the plugin:
streams [env: TAP_REST_API_MSDK_STREAMS] current value: [{'name': 'current_weather', 'path': '/onecall', 'params': {'lat': 52, 'lon': 13, 'cnt': 7, 'appid': '$OPENWEATHER_KEY', 'exclude': 'hourly,current,minutely,alerts'}, 'primary_keys': ['dt']}] (from `meltano.yml`)
v
Specify it as a setting in the metadata, and then you can use all the "standard" stuff
j
I add the appid in the settings 😉
And it shows up correctly in another list command.
But is not applied in the streams dict.
v
What is "correct" to you here?
Copy code
'appid': '$OPENWEATHER_KEY'
Should be what?
(Not trying to get your key, but you want that go be the key not the $ pointer?
j
A random alphanumeric string pulled from my 
.env
file.
Correct.
v
https://docs.meltano.com/guide/configuration#configuring-settings Is worth a read I think as you can do pretty much whatever you want. Easiest thing here would probably be 
export TAP_REST_API_MDK_APPID='abcdefg'
j
Hi Derek, been there and tried it. Now I have the 
TAP_REST_API_MDK_APPID
. Still not being applied in the stream.
Copy code
- name: tap-rest-api-msdk
  pip_url: tap-rest-api-msdk
  executable: tap-rest-api-msdk
  namespace: tap_rest_api_msdk
  capabilities:
    - state
    - catalog
    - discover
  settings:
    - name: appid
  config:
    api_url: <https://api.openweathermap.org/data/2.5>
    streams:
      - name: current_weather
        path: /onecall
        params:
          lat: 52
          lon: 13
          cnt: 7
          appid: $TAP_REST_API_MSDK_APPID
          exclude: hourly,current,minutely,alerts
        primary_keys:
          - dt
Yields again, just the verbose value for the appid, not the actual key.
Hence my question.
v
Copy code
settings:
- name: appid
    config:
      api_url: <https://api.openweathermap.org/data/2.5>
      streams:
        - name: current_weather
          path: /onecall
          params:
            lat: 52
            lon: 13
            cnt: 7
            appid: $TAP_REST_API_MSDK_APPID
            exclude: hourly,current,minutely,alerts
          primary_keys:
            - dt
What is this from? doesn't look like valid 
meltano.yml
to me
j
This is from my 
meltano.yml
and the indentation got mixed up on copying over here.
Other than that it compiles perfectly fine to the tap’s config.
v
can you post with proper indenting
j
edited above message
v
Can you post your entire 
meltano.yml
?
j
Happily 🙂
Copy code
version: 1
default_environment: dev
project_id: foobar
plugins:
  extractors:
  - name: tap-rest-api-msdk
    pip_url: tap-rest-api-msdk
    executable: tap-rest-api-msdk
    namespace: tap_rest_api_msdk
    capabilities:
      - state
      - catalog
      - discover
    settings:
      - name: api_url
      - name: auth_method
      - name: auth_token
      - name: name
      - name: path
      - name: params
      - name: headers
      - name: records_path
      - name: primary_keys
      - name: replication_key
      - name: except_keys
      - name: num_inference_records
      - name: appid
    config:
      api_url: <https://api.openweathermap.org/data/2.5>
      streams:
        - name: current_weather
          path: /onecall
          params:
            lat: 52
            lon: 13
            cnt: 7
            appid: $TAP_REST_API_MSDK_APPID
            exclude: hourly,current,minutely,alerts
          primary_keys:
            - dt
  loaders:
  - name: target-csv
    variant: hotgluexyz
    pip_url: git+<https://github.com/hotgluexyz/target-csv.git@0.3.3>
  - name: target-postgres
    variant: transferwise
    pip_url: git+<https://github.com/transferwise/pipelinewise-target-postgres.git>
    config:
      host: $TARGET_POSTGRES_HOST
      port: $TARGET_POSTGRES_PORT
      user: $TARGET_POSTGRES_USER
      password: $TARGET_POSTGRES_PASSWORD
      dbname: $TARGET_POSTGRES_DBNAME
      default_target_schema: $TARGET_POSTGRES_SCHEMA

environments:
- name: dev
- name: staging
- name: prod
v
Helps a ton ➕
Ok so to reword you question a bit as this is a strange tap configuration I think it'd be someting like Can I reference an environment variable inside of a configuration that is a json object? If you were to dump the config here you'd see the object, this tap is odd as you are doing a ton of configuration work in the 
streams
config. One option (although not great) although it'd answer your original question would be 
meltano config tap-clickup set streams "{"params":["appid":"abcdefg"]" --store dotenv
(Or set the same variable directly as an environment variable instead which also works)
Someone else may have another option, I think there's an issue in somewhere about this as well for a nested variable being expanded. I'm not sure if it is / isn't doable
@jan_kyri another option for you is to use a tap designed for that API https://hub.meltano.com/taps/openweathermap
j
Hi @visch thanks for digging into this so deeply. New to Meltano and the Openweathermap was just a dummy API choice to test general functionality. Helps to read your perspective on this specific’s tap implementation and the reference. I’ll take a look at Openweathermap and see if I can work off of that. Thank you, much appreciated! 🙏
v
No problem, welcome to the community!
r
meltano config tap-clickup set streams "{"params":["appid":"abcdefg"]" --store dotenv
Did you accidentally use a different tap name here?
v
yes I just through a name in 😄 any name works fine so 
tap-rest-api-msdk
in this case
4 Views
Open in Slack
PreviousNext
Powered by