Hello all, I am trying to understand settings of tap-RestAPI (tap-rest-api-msdk), there is an API that I need to use as tap. I have tested the API on Postman and it’s working fine, I have set the same parameters, header etc. but I am getting “KeyError: ‘streams’” error. I tried to set the headers(Content-Type and Auth) at the same time because there is no way to set 2 separate headers in a different variables. I have shared my settings and the API link, I am open to wise suggestions 🙂 And also, normally we’re saving the tap secrets in .env file but there is no option to save API token? Lumoa API - Get Feedback = https://documenter.getpostman.com/view/2165848/lumoa-feedback-api/7EBhFfY

Looks like the tap requires a `streams` config option, but this isn't documented on the hub. https://github.com/Widen/tap-rest-api-msdk/blob/3585b2553adbb8463c8b5d4cf1494d46988981fa/tap_rest_api_msdk/tap.py#L161-L170 https://github.com/Widen/tap-rest-api-msdk/blob/3585b2553adbb8463c8b5d4cf1494d46988981fa/tap_rest_api_msdk/tap.py#L184-L242

@nuh_mehmet_kaya I havent used this tap personally but a couple things I noticed that might help. • The headers config is an object and it looks like headers are a key/value pairs instead of a string and no need for

--headers

in there • Related to the api key you can using templating to set it in the .env file and read it into a setting

${API_TOKEN}

(although I dont think object settings support templating yet 😢 - relevant issues https://github.com/meltano/meltano/issues/3171 and https://github.com/meltano/meltano/issues/3435 )

I think the idea is that

streams

is a set of per-stream overrides for the top-level properties (

records_path

,

except_keys

,

path

, etc). If you don't need that functionality, you may have to set it to an empty array (

[]

) if the tap requires it.

I created https://github.com/meltano/hub/issues/1056 to make sure we document this tap better on the hub later.

Hi, @Reuben (Matatika), @pat_nadolny Thanks for the answers, I will try to make the changes and try again. Do we have any other tap options to get data from an API?

In a generic way for REST APIs, no - this is the only one currently. Have you looked into creating a tap for this API using the Meltano SDK? If you have the time, it would definitely be worth a look into.

Hi @Reuben (Matatika), I haven’t checked this before, is it easy to create our own taps?

In my opinion, yes - the docs are good, there's loads of example, and people in here are very helpful when it comes to SDK questions. There are some Python concepts that are fundamental to using the SDK that would be helpful to know beforehand (namely inheritance and method overriding).