I am very confused by a simple problem I have been...
# singer-taps
m
I am very confused by a simple problem I have been suck on for a while now. Within my meltano.yaml I am refrencing env variable fine within the loaders and targets, liek this:
Copy code
- name: tap-sistrix
    inherit_from: tap-rest-api-msdk
    config:
      streams:
      - name: sistrix_${TAP_SISTRIX_ACCOUNT_ID}_${TAP_RUN_DATE}
      params:
        api_key: ${TAP_SISTRIX_API_KEY}
        format: json
        date: ${TAP_SISTRIX_DATE}
        domain: ${TAP_SISTRIX_DOMAIN}
but when I try and do the same in the mappers config block the env do not get replaced. The example I have is using th google search console tap:
Copy code
- name: rename_search_tables
      config:
        stream_maps:
          performance_report_custom:
            __alias__: search_console_performance_custom_${TAP_SEARCH_ACCOUNT_ID}
The output stream name is just search_console_performance_custom_${TAP_SEARCH_ACCOUNT_ID} rather than the wanted table name with env. Any ideas why it works in one spot of the .yaml and not the other?
t
This a missing feature of mappers/stream maps unfortunately. Will you give this issue a thumbs up please? https://github.com/meltano/meltano/issues/7151
m
Damn. Have given that a thumbs up. Any alternative ways of going about this?
t
@Edgar Ramírez (Arch.dev) do you know any work arounds?
e
I think you could use the tap's stream map config instead of the mapping's, but let me confirm it.
Alright, so that should work and
tap-rest-api-msdk
should support it since it's SDK-based:
Copy code
- name: tap-sistrix
  inherit_from: tap-rest-api-msdk
  config:
    streams:
    - name: sistrix_${TAP_SISTRIX_ACCOUNT_ID}_${TAP_RUN_DATE}
    params:
      api_key: ${TAP_SISTRIX_API_KEY}
      format: json
      date: ${TAP_SISTRIX_DATE}
      domain: ${TAP_SISTRIX_DOMAIN}
    stream_maps:
      config:
        stream_maps:
          performance_report_custom:
            __alias__: search_console_performance_custom_${TAP_SEARCH_ACCOUNT_ID}
m
sorry the actual tap I am struggling with wAs tap-google-search-console but I guess if that is also SDK (not sure how to check that) based same principle?
Looking at the doc page..no the singer variant i am using does not look to use the SDK. I guess I could switch to the airbyte version which does and use the above method https://hub.meltano.com/extractors/tap-google-search-console/
e
Yeah, that's unfortunately not SDK-based. Let me think of something else...