Hi, I want to ask you if there is a way to downloa...
# plugins-general
j
Hi, I want to ask you if there is a way to download data using tap-rest-api-msdk extractor and manipulate dataset? Precisely, I need to include one column extra with flag what ID I refer to. I just noticed that my incrementation is missing one column. Meltano.yaml
Copy code
records_path: $.data[*]
p
Can you share more? What commands are you running? I dont think I understand what youre trying to do yet 😄. You might be able to use mappers to add data or do minor transformations inline between the tap/target
j
Hi @pat_nadolny I am using meltano to fetch and load data into postgres database. My problem is that I want to add one column into data with ID from meltano.yml file (which I generate with python script).
Copy code
- name: elabjournal_sections_meta
  inherit_from: tap-rest-api-msdk
  config:
    streams:
    - name: elab_sections_meta
      path: /experiments/sections/{{ expJournalID }}/meta
      headers:
        Authorization:  {{ ELAB_API_KEY }}
      primary_keys:
      - expJournalMetaID
      records_path: $.data[*]
      params:
        projectID: {{ID}}
I need to add extra column to dataset for exaple: expJournalID = 1,2,3,4, etc
Could you please help me to configure this yaml file? Thank you in advance
p
Oh yeah you should be able to use mappers for that
Copy code
config:
      stream_maps:
        elab_sections_meta:
          new_column: 'your_static_value'
I think that might work but you'd have to test it
it adds a
new_column
field thats populated with
your_static_value
for each record to your
elab_sections_meta
stream
j
alright, so those mappers are configured separately? Right now I use 4 extractors and loader
p
Theres 2 types of mapper functionality. All SDK taps/targets have them built in so you can configure them in either the tap or target. OR you can use a standalone mapper that goes between a tap and a target
tap mapper target
You should be able to configure it once on your loader in this case
j
ok, thank you so much for help