Hello Is it possible to change meltano parameters for elt on Meltano #getting-started

Hello, Is it possible to change meltano parameters...

thomas_pedot

01/13/2023, 5:09 PM

Hello, Is it possible to change meltano parameters for elt on the fly ? I am using dagster as a pipeline manager and I need to change plugins.extractor.config.streams[0].params without changing the yaml file it self. It runs with this command line meltano elt tap-rest-api-msdk target-jsonl

thomas_pedot

01/13/2023, 5:17 PM

Copy code

config:
      api_url: ...
      streams:
      - name: ...
        path: ...
        params:
          date_after: 01/01/2021

thomas_pedot

01/13/2023, 5:17 PM

I want to override date_after

Matt Menzenski

01/13/2023, 5:28 PM

I believe you could provide the entire

streams

value via a

TAP_REST_API_MSDK_STREAMS

environment variable as a JSON string, you could provide that with an updated date_after value but you’d need to provide the entire streams object to do it

thomas_pedot

01/13/2023, 5:33 PM

I will look at it. The problem I might face is that dagester (the run that will launch the program) is loaded already. So variable or parameter injection via CLI may be better.

thomas_pedot

01/14/2023, 12:47 PM

Hello, I open an issue and a PR in dagster-meltano for my full use case. If you have any comment, please do 🙂 @Matt Menzenski I finally end with your solution with TAP_REST_API_MSDK_PARAMS. Thanks for helping me understand meltano better https://github.com/quantile-development/dagster-meltano/pull/20

Open in Slack

Previous Next