Hello! 👋 I’m using the tap-github and the job fails because of Github API rate limit:

RuntimeError: All GitHub tokens have hit their rate limit. Stopping here

I tried to use the config additional_auth_token, the runtime lasts longer and I can see that my tokens have been used but in the end, the job still fails. I even tried 8 additional tokens. Should I try more additional tokens (which leads to a long solution) or someone has a hinch on how to debug Github rate limit? Thanks a lot!

Haven't messed with the github tap myself but have you tried moving your start date up? Ideally state would be saved for you so eventually the tap would catch up if you keep running it

I’m quite new into data engineering so as far as I know, I already have a table full of Github data that I daily update but one day, the job failed because of the rate limit. As my start_date hasn’t changed since I create the job, you mean that my job gets all the data from the same start_date until the time it’s triggered (that explains the increasing amount of data collected through the API). I’m not very familiar with the state notion: at every run, the state is stored and passed to the next run to start updating only since the state? Many thanks for your help!! 🙂

you mean that my job gets all the data from the same start_date until the time it’s triggered (that explains the increasing amount of data collected through the API)

I’m not very familiar with the state notion: at every run, the state is stored and passed to the next run to start updating only since the state?

https://hub.meltano.com/singer/spec#state

> increasing amount of data collected through the API)

I'm not sure what this means exactly. Are you pulling the same data over and over again and that's the issue? Is state being updated between each run? What do you expect to happen vs what are you seeing? Basically you need to dive in a bit more to understand what's going on