Do scraping APIs charge for a 404 or a 429?
Usually yes for a 404: all 7 web scraping APIs whose billing rule names the status charge for it (Firecrawl, Scrape.do, ScrapeOps, ScrapeOwl, ScraperAPI, ScrapingBee and ZenRows), because the target site answered and the API delivered its reply. A 429 goes the other way: 5 rules leave it out of what they charge, 2 by name (ScrapeOwl and ScrapingBot) and 3 because they charge only the statuses they list (Scrape.do, ScraperAPI and ScrapingBee). A 403 from bot protection is free on 4 of the 5 rules that cover it; Firecrawl charges it. A request stopped by bot protection or a CAPTCHA is free on all 8 rules that name it or leave it off their list; ScrapingBot and ScrapeBadger also refund a timeout, while ScraperAPI charges a request you cancel before its 70 seconds are up.
Worked out from each vendor's own pricing page; each vendor page on this site shows the day its price was read. Only the statuses a vendor names are counted; 23 web scraping APIs bill successes only without listing statuses.| API | 404: page not found | 403: blocked | 429: too many requests | 500 or another 5xx: the site's server failed | Stopped by bot protection or a CAPTCHA | Timed out | The vendor's rule |
|---|---|---|---|---|---|---|---|
| Firecrawl | Charged | Charged | Not named | Not named | Not named | Not named | error status such as 403 or 404 is still returned to you and costs 1 credit |
| Scrape.do | Charged | Not charged (not on its list) | Not charged (not on its list) | Not charged (not on its list) | Not charged | Not named | Only 2xx, 400, 404 and 410 responses consume credits. Other failed or blocked responses are not charged |
| ScrapeOps | Charged | Not named | Not named | Not named | Not named | Not named | returns a successful status code 200 or 404 consumes API credits |
| ScrapeOwl | Charged | Not named | Not charged | Not named | Not named | Not named | HTTP 200 and HTTP 404 responses consume credits |
| ScraperAPI | Charged | Not charged (not on its list) | Not charged (not on its list) | Not charged (not on its list) | Not charged (not on its list) | Charged if you cancel it before 70 seconds | Only HTTP 200 and 404 responses are charged. A client-cancelled request is also charged if cancelled before ScraperAPI has had 70 seconds to finish |
| ScrapingBee | Charged | Not charged (not on its list) | Not charged (not on its list) | Not charged (not on its list) | Not charged (not on its list) | Not named | We only charge for successful requests, i.e returning with a 200, 404 or 410 status code |
| ScrapingBot | Not named | Not charged | Not charged | Not charged | Not charged | Not charged | Refunded automatically: timeouts, blocks and other 4xx answers such as 401, 403 and 429, any 5xx from the target |
| ZenRows | Charged | Not named | Not named | Not named | Not named | Not named | 404 or 410 responses count as usable results |
| Crawlbase | Not named | Not named | Not named | Not named | Not charged | Not named | A request that fails or is blocked costs nothing |
| HasData | Not named | Not named | Not named | Not named | Not charged | Not named | failed or blocked requests do not cost credits |
| ScrapingAnt | Not named | Not named | Not named | Not named | Not charged | Not named | Blocked and error responses are not billed |
| ScrapeBadger | Not named | Not named | Not named | Not named | Not charged | Not charged | Failed requests (blocks, timeouts, errors or empty responses) are never charged |
A 404 you did not expect usually means a stale URL list, and every one of them is a paid request on these APIs, so clean the list before a large crawl. A 429 is a signal to slow down: a retry costs nothing where the rule leaves 429 uncharged, but it can fail again and still takes up your concurrency.
Why a 404 is usually a paid request
“We only charge for successful requests” sounds as if every error is free. For most of these APIs, success means the API did its job: it reached the target and brought back the target’s answer. A 404 is the target’s answer, so the request counts. What is free is the API failing on its own side, or the target blocking it: a timeout, a CAPTCHA wall, a 403 from bot protection, a 5xx from the API itself. Firecrawl is stricter and charges for any page it returns, a 403 from the target included.
Why a 429 is not
A 429 means “too many requests”: the target, or the API’s own limits, refused the request because you sent too many too fast. The vendors that name it treat it as a failure to deliver, not a delivered page, so it costs nothing, and the vendors that charge only the statuses they list do not list it. It still costs time, and a crawl that hits 429s needs a lower request rate or the higher concurrency of a bigger plan, which each vendor page lists under “Limits and terms”.
How the table is read
Each row comes from the vendor’s own billing rule as recorded on its page on this site, quoted word for word; a status the vendor does not name is shown as not named, never guessed. The three statuses with the most published rules have their own pages, with what causes each and how to stop it: 403 Forbidden, 404 Not Found and 429 Too Many Requests. To see what any failure rate does to each bill, try the failed requests tool.