Thank you! This actually works perfectly for me! I...
# plugins-general
j
Thank you! This actually works perfectly for me! I have last question, I need a nested endpoint to fetch data based on previous endpoint result. Do you know how to parametrize my meltano.yml file? precisely AAA
Copy code
path1: path: /exp
path2: path: /exp/AAAAAA/sec
Copy code
version: 1
default_environment: prod
project_id: XXX
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-rest-api-msdk
    variant: widen
    pip_url: tap-rest-api-msdk
    config:
      api_url: <https://company.restapi.com/api/v1>
      streams:
      - name: elab_exp
        path: /exp
        headers:
          Authorization: XXX
        primary_keys:
        - expID
        records_path: $.data[*]
        params:
          projectID: 00

      - name: elab_sec
        path: /exp/AAAAAA/sec
        headers:
          Authorization: XXX
        primary_keys:
          - sectID
        records_path: $.data[*]
        params:
          projectID: 00
a
Sounds like a parent-child stream? https://sdk.meltano.com/en/latest/parent_streams.html But you would need to build a custom extractor to use the SDK functionality, I'm not sure the generic rest-api-msdk team supports that. I'm assuming you would know the value(s) to replace 'AAAAA' with from the initial stream run? And it would be dynamic, more than just a few set values?
j
Yes, exactly that's what I need. I need to fetch the data elab_sec based on a previous values from endpoint exp. Is it possible to run it with a list of values? Or maybe thrpugh cmd line?
Or the other way - I have already a value of AAAA, is there a way to rerun it with this value multiple times?
e
I don't think there's a way to do what you need with
tap-rest-api-msdk
, so like Andy said, you need to create a custom extractor: https://docs.meltano.com/tutorials/custom-extractor