Home / Scraping errors / 404 Not Found
404 Not Found when scraping: is it billed?
Yes, a 404 is a paid request on every one of the 7 web scraping APIs whose billing rule names it (Firecrawl, Scrape.do, ScrapeOps, ScrapeOwl, ScraperAPI, ScrapingBee and ZenRows): the site answered, and the API delivered its answer. So 10,000 dead links in a 100,000-page month cost from $1.90 to $9.90 at each API's own plain-page price, the price of a list that was not cleaned first.
Each rule is taken from the vendor's own billing documentation as recorded on its page here; each rate is worked out from its own pricing page for a 100,000-page month.| API | A 404 is | 10,000 dead links cost | The vendor's rule |
|---|---|---|---|
| ScrapeOps | Charged | $1.90 | returns a successful status code 200 or 404 consumes API credits |
| Scrape.do | Charged | $2.90 | Only 2xx, 400, 404 and 410 responses consume credits. Other failed or blocked responses are not charged |
| ScrapeOwl | Charged | $2.90 | HTTP 200 and HTTP 404 responses consume credits |
| ZenRows | Charged | $3.90 | 404 or 410 responses count as usable results |
| ScraperAPI | Charged | $4.90 | Only HTTP 200 and 404 responses are charged. A client-cancelled request is also charged if cancelled before ScraperAPI has had 70 seconds to finish |
| ScrapingBee | Charged | $4.90 | We only charge for successful requests, i.e returning with a 200, 404 or 410 status code |
| Firecrawl | Charged | $9.90 | error status such as 403 or 404 is still returned to you and costs 1 credit |
| ScrapingAnt | Not named | not named | Blocked and error responses are not billed |
| ScrapeBadger | Not named | not named | Failed requests (blocks, timeouts, errors or empty responses) are never charged |
| ScrapingBot | Not named | not named | Refunded automatically: timeouts, blocks and other 4xx answers such as 401, 403 and 429, any 5xx from the target |
| HasData | Not named | not named | failed or blocked requests do not cost credits |
| Crawlbase | Not named | not named | A request that fails or is blocked costs nothing |
A retry does not fix a 404 and costs another request; check the address instead. Some sites answer a blocked request with a 404 page, so a sudden run of 404s on pages that load in a browser is worth testing with a protected mode before deleting the URLs.
Why a 404 counts as a paid request
A 404 means the site looked for the page and found nothing at that address. For a scraping API that is a delivered answer: it reached the site, got past whatever protection stood in the way and brought back the site’s reply. Every API that names the 404 in its billing rule charges it, because from its side the job was done. “We only charge for successful requests” means successful for the API, not for you.
Where 404s come from
Most 404s in a scraping job come from the URL list, not the site:
- Product, listing and job pages removed since the list was built; on shops and job boards a list can lose a large share of its pages within weeks.
- URLs built by hand or by a pattern that does not hold for every item.
- Encoding mistakes: a space, an ampersand or a non-English character not escaped in the address passed to the API.
- Links collected from an old sitemap or a third-party list.
How to keep them off the bill
Refresh the URL list from the site’s current sitemap or category pages before a large crawl, and drop addresses that returned a 404 last time instead of retrying them. A retry never turns a 404 into a page, and on these APIs each retry is charged again. If a whole batch suddenly returns 404s for pages that open in your browser, test a few through a protected mode first: some sites answer a blocked request with a 404 page rather than a 403.
The URL encoder escapes a target address for an API’s url parameter, and the URL parser normalises a list before a crawl.