Meltano

image.png

also running into rate limiting issues, is there a way to slow down the frequency of requests made? What’s recommended for pausing &amp; waiting / re-running (or is this the goal of the state JSON, so duplicate data pulls are avoided?)

Screen Shot 2020-08-07 at 11.09.41 AM

&gt;  is there a way to slow down the frequency of requests made?
<@U06C5J96CH4> Not from Meltano's level, since all of the requests to Slack are handled by the tap executable, which Meltano has little say over beyond the flags it passes. If `tap-slack`  isn't handling rate limiting correctly, that'd need to be addressed inside `tap-slack`

&gt;  What’s recommended for pausing &amp; waiting / re-running (or is this the goal of the state JSON, so duplicate data pulls are avoided?)
You can re-run as much as you like, because of the state, but that's obviously not a great strategy for dealing with rate limiting

right better to have the tap handle rate-limiting logic better

the Singer slack has a tap-slack channel, so I’m also seeing how they think about this / if theres more we can do with the rate-limiting logic

based on my interpretation of teh logs, meltano is getting rate limited, backs off

and there’s some success messages in between failure messages so

Judging by those logs you sent, it looks like it's already handling rate limiting pretty well

yuppp the maintainer of that tap said that everything was fine (and my ELT ended up finishing successfully)

he said he’d update the log messages to “look” less worrisome :wink:

now to try this all with amazon RDS instead of just local JSON…