https://linen.dev logo
Join Slack
Powered by
For one of our users we'd like to offer the simple...
# best-practices
r
For one of our users we'd like to offer the simplest path to fetching data from a custom REST endpoint that has strict rate limits. We're considering rolling a tap with https://pypi.org/project/ratelimit/ but also found https://github.com/Widen/tap-rest-api-msdk What do you guys do for this?
v
Honestly the easiest way forward imo is to make a new tap with the singer sdk, it's really really simple especially if you only have one endpoint.
I am biased against general tap-rest-api taps though so 🤷 I just see so many bad things happen when people use them
Don't get me wrong they have their place but I'd say 90% of the time it should just be a tap
r
This client isn't too keen on rolling their own because their view is: I've hand rolled many REST-api-like connectors and they all sort of share the same rate limiting characteristics and I'm really looking for an abstraction that lets me forget about rate limits, if it exists. Rolling a generic one a new singer tap would basically be reimplementing tap-rest-api-msdk. Then why not just extend that tap with rate limiting capabilities (if it doesn't have it) instead of reinventing the wheel?
I just see so many bad things happen when people use them
Could you give one or two examples such that I can try to steer them clear of making those mistakes?
(Thanks for responding by the way 🙂!)
v
That's exactly what the use case is for the SDK, they don't have a rate limiting feature added yet but it's on the docket (It's pretty easy to do though see https://github.com/AutoIDM/tap-zohosprints/blob/main/tap_zohosprints/client.py#L117 another here https://github.com/AutoIDM/tap-clickup/blob/main/tap_clickup/client.py#L72 They tend to be fairly custom. MR is in for adding some generic rate limits here https://gitlab.com/meltano/sdk/-/merge_requests/277
The SDK is itself a generic Rest connector tbh, it's going to do way better than a tap can. Again this is opinionated we are in the right channel for this though!
Let me see there was a great example here it is https://meltano.slack.com/archives/C01TCRBBJD7/p1650902572839519 read through this thread
r
tap-rest-api-msdk
Is also built with the Meltano SDK right?
v
Yes
r
make a new tap with the singer sdk
So you're saying roll a tap per REST-endpoint you want to connect to?
v
Yes
r
I guess once https://gitlab.com/meltano/sdk/-/merge_requests/277 is in that makes a bit more sense
v
In a past life I tried to go down the path of those two examples you gave. Ultimately what you end up doing is coding in configuration
So you take all the code you would have written and slap it into configuration
I'll stop chatting though generally as some other opinions should be able to come offset me!
Ok one last point as I'm already on a roll. Each API I've ever dealt with is unique. Which seems odd, "they are all just rest apis they are the same in some ways right", no they are not. Each has it's own value which is why the API is valuable to use to pull data out of, then on top of that "rest apis" are generally RPCs not strictly Rest (but I"ll leave that be) and everyone implements everything differently for different (and sometimes good) reasons. Even just using Google's APIs which are the most standard I've ever seen across products they vary a lot even when you consider using their ProtoBufs and all of that.
r
I hear you! I figured the "declarative approach configure REST tap once for rate limits and endpoint/entities once" is a pipe dream (because of minor differences in REST endpoints that are like you say more like RPCs) I'll have a chat with that user to see if 
tap-rest-api-msdk
+ rate limits (we could PR to that tap) would work for his use case (it's a proprietary e-commerce API).
e
d
@edgar_ramirez_mondragon Is https://gitlab.com/meltano/sdk/-/merge_requests/277 ready for another round of review?
e
https://gitlab.com/meltano/sdk/-/merge_requests/277 was merged earlier today 🙂
2 Views
Open in Slack
PreviousNext
Powered by