https://linen.dev logo
Join Slack
Powered by
Helllo <@U06D0USC2CQ> I am facing rate limiter er...
# troubleshooting
j
Helllo @Reuben (Matatika) I am facing rate limiter error for tap-facebook -- any insights about how to handle that. Sharing error for reference, Backing off 128.40 seconds after 7 tries calling function <bound method HTTPStream.request of <tap_facebook.streams.ads.AdsStream object at > with args (<PreparedRequest [GET]>, None) and kwargs {} cmd_type=elb consumer=False job_name=integration:tap-facebook-to-target-postgres name=tap-facebook producer=True
a
Is your App still in developer mode? https://github.com/MeltanoLabs/tap-facebook/issues/88
I'm not familiar with the tap, and this is an older issue, but just trying to help if I can!
➕ 1
r
Same here - haven't used the tap so can't really comment. Give a thumbs up or a comment on the issue would be my suggestion, if Andy's tip about app developer mode does not help resolve.
j
Hello Andy, Thanks for the response. No my app is in live mode
e
Any insights about how the API communicates usage and reset is appreciated. PRs are even better 😄
n
I am using airflow so i can write python around it but here's a little of what i am doing. I have corporate accounts and accounts under the corporation. Facebook sends back headers with info. I run this check before a load a new batch. As far as rate limits it really depends on the account, it varies by number of ads and campaigns. More of those higher the rate limit.
Copy code
def check_rate_limit(account_number, **kwargs):
    """Check rate limits for an account. If limits exceed 50%, trigger a retry."""
    headers = {"Authorization": f"Bearer {FACEBOOK_ACCESS_TOKEN}"}
    url = f"<https://graph.facebook.com/v20.0/act_{account_number}>" if account_number != "me" else "<https://graph.facebook.com/v20.0/me>"

    response = requests.get(url, headers=headers)
    if response.status_code == 200:
        app_usage = json.loads(response.headers.get("X-App-Usage", "{}"))
        business_usage = json.loads(response.headers.get("x-business-use-case-usage", "{}"))

        call_count = app_usage.get("call_count", 0)
        total_cputime = app_usage.get("total_cputime", 0)
        total_time = app_usage.get("total_time", 0)

        account_usage = business_usage.get(str(account_number), [{}])
        if isinstance(account_usage, list) and account_usage:
            account_usage = account_usage[0]
        else:
            account_usage = {}

        account_call_count = account_usage.get("call_count", 0)
        account_cputime = account_usage.get("total_cputime", 0)
        account_total_time = account_usage.get("total_time", 0)

        print(f"\nRate Limits for {account_number}:")
        print(f" App-Level Call Count: {call_count}%")
        print(f" App-Level CPU Time: {total_cputime}%")
        print(f" App-Level Total Time: {total_time}%")
        print(f" Account-Level Call Count: {account_call_count}%")
        print(f" Account-Level CPU Time: {account_cputime}%")
        print(f" Account-Level Total Time: {account_total_time}%")

        if (
            call_count > 50 or total_cputime > 50 or total_time > 50 or
            account_call_count > 50 or account_cputime > 50 or account_total_time > 50
        ):
j
Hello All, Thanks for the response. Could this be because of older version, Getting this error 17 with subcode 2446079 Indicates that the token being used in the Ads API v3.3 or older request has reached its rate limit.
@Edgar Ramírez (Arch.dev) I can help you with this -- This is my yml configurations for tap-facebook - name: tap-facebook variant: meltanolabs pip_url: git+https://github.com/MeltanoLabs/tap-facebook.git config: account_id: start_date: end_date: api_version: access_token: use_batch_api: true batch_size: 10 # Reduce this further from 20 page_size: 25 # Add this to reduce records per page request_timeout: 600 api_request_rate_limit: 1 api_request_rate_limit_window: 60 # One request per minute (more conservative) max_backoff: 3600 # 1 hour max backoff max_attribution_windows: 1 # Limit attribution windows if applicable include_deleted: false stream_maps: adaccounts: timezone_offset_hours_utc: "int(timezone_offset_hours_utc)" The strange thing on my dashboard I can't see any throttle issues - my application rate limitshows 100% left. However on the contrary, I get these error when I try to run tap-facebook INFO | backoff | Backing off _request(...) for 128.8s (singer_sdk.exceptions.RetriableAPIError: 400 Client Error: b'{"error":{"message":"User request limit reached","type":"OAuthException","is_transient":false,"code":17,"error_subcode":2446079,"error_user_title":"Ad account has too many API calls","error_user_msg":"There have been too many calls from this ad account. Please wait a bit and try again.","fbtrace_id":"A8koBa1sWpE8zEN3foNcqYq"}}' (Reason: Bad Request) for path: ) cmd_type=elb consumer=False job_name=integration:tap-facebook-to-target-postgres name=tap-facebook producer=True stdio=stderr string_id=tap-facebook | ERROR | root | Backing off 128.80 seconds after 7 tries calling function <bound method _HTTPStream._request of <tap_facebook.streams.ads.AdsStream object at 0x76c4986e67b0>> with args (<PreparedRequest [GET]>, None) and kwargs {} cmd_type=elb consumer=False job_name=integration:tap-facebook-to-target-postgres name=tap-facebook producer=True run_id= stdio=stderr string_id=tap-facebook
n
@Jack Sparrow The rate limits "page" on facebook does not take into account the cpuTime rate limit. I found that i will hit that way before i hit my actual rate limits for call counts.
❤️ 1
👀 1
j
Makes sense - can you please share how you tackled this issue?
n
I wrote some python to check before each batch call i make to see whether or not i need to back off. It's in this thread about 3 comments up. I use Airflow for my orchestrator and this is in one of my DAGs You can test it by running your pull till it fails or just run it a little. then fire off a curl https://graph.facebook.com/v20.0/act_{account_number}" and you can see the results in the header
✅ 1
6 Views
Open in Slack
PreviousNext
Powered by