Home / Scraping errors / 429 Too Many Requests
429 Too Many Requests when scraping: what it costs and how to stop it
A 429 means requests arrived too fast, either for the site you are scraping or for your plan's limit on requests at once. It costs nothing on all 5 scraping APIs whose billing rule covers it, so a retry is free there; what it costs is time. Your plan sets the pace: about $100 a month buys 50 requests at once on the middle web scraping API.
Each rule is taken from the vendor's own billing documentation as recorded on its page here; each rate is worked out from its own pricing page for a 100,000-page month. Concurrency is the number of requests a plan lets you have in flight at the same time.| API | A 429 is | Smallest paid plan | Plan nearest $100 |
|---|---|---|---|
| ScrapeOps | Not named | 1 on 25,000 credits, $9 | 50 on 1,250,000 credits, $99 |
| ScrapingFish | Not named | 1 on Unlimited monthly, concurrency 1, $100 | 1 on Unlimited monthly, concurrency 1, $100 |
| Spider.cloud | Not named | 2 on Unlimited 2 concurrency, $40 | no plan between $75 and $125 |
| ScrapeGraphAI | Not named | 3 on Starter, $20 | 15 on Growth, $100 |
| BrowserCloud | Not named | 5 on Hobby, $19 | 50 on Startup, $99 |
| Firecrawl | Not named | 5 on Hobby, $19 | 25 on Standard, $99 |
| HasData | Not named | 5 on Startup, $59 | 15 on Basic, $119 |
| Scrapfly | Not named | 5 on Discovery, $30 | 20 on Pro, $100 |
| ScrapingDog | Not named | 5 on Lite, $40 | 50 on Standard, $90 |
| Scrape.do | Not charged (not on its list) | 10 on Hobby, $29 | 50 on Pro, $99 |
| ScrapeOwl | Not charged | 10 on Bootstrap, $29 | 25 on Startup, $99 |
| ScrapingBot | Not charged | 10 on Starter, $49.99 | 50 on Startup, $99.99 |
| WebScraping.AI | Not named | 10 on Personal, $29 | 25 on Plus, $99 |
| ScraperAPI | Not charged (not on its list) | 20 on Hobby, $49 | no plan between $75 and $125 |
| WebScrapingAPI | Not named | 20 on Starter, $19 | 50 on Standard, $99 |
| ZenRows | Not named | 20 on Build B1, $19 | 50 on Launch L2, $99 |
| Hyperbrowser | Not named | 25 on Startup, $30 | 100 on Scale, $100 |
| Lightpanda Cloud | Not named | 30 on Builder, $19 | no plan between $75 and $125 |
| ScrapingBee | Not charged (not on its list) | 50 on Freelance, $49 | 100 on Startup, $99 |
| Desearch | Not named | 150 on Starter, $179 | no plan between $75 and $125 |
| Olostep | Not named | 150 on Starter, $9 | 500 on Standard, $99 |
A 429 from the API itself means your scraper sent more requests at once than the plan allows: queue requests to the plan's limit instead of retrying at once. A 429 the API passes on from the site means the site is rate-limiting; a run of them calls for fewer requests per second to that site, or a mode that spreads the requests over more addresses.
Two different 429s
A 429 means “slow down”. When you scrape through an API it can come from two places, and the fix differs:
- From the API itself. Your scraper sent more requests at once than your plan allows. Every plan has a concurrency limit, the requests it will run at the same time; on a plan that allows 50, request number 51 is refused, usually with a 429, until one of the 50 finishes.
- From the site, passed on by the API. The site you are scraping decided too many requests arrived from one address or one session. An API that rotates proxies spreads your requests over many addresses, so you see this mostly when a site is very strict or your requests all go to one small site.
What it costs
On the APIs that name the 429 in their billing rule, a refused request is free (the table above quotes each rule), so it costs nothing but the time to send it again. The real cost is the plan: if your job needs more requests at once than the plan allows, you either wait or move to a bigger plan, and on many APIs the concurrency rises with each plan whether or not you need the extra credits.
How to stop getting it
For a 429 from the API, cap your scraper’s open requests at the plan’s limit and queue the rest, rather than sending everything and retrying the refusals; most HTTP libraries and scraping frameworks have a concurrency setting for this. For a 429 from the site, send fewer requests per second to that site, spread a crawl over more hours, and wait before retrying: as long as the response’s Retry-After header asks when it carries one, otherwise doubling the wait each time. The table above gives each API’s limit on its smallest plan and on the plan nearest $100 a month; the rate limit page lists the per-second and per-minute caps vendors publish.