Home / Scraping errors / 403 Forbidden
403 Forbidden when scraping: what it costs and how to get past it
A 403 when you scrape usually means the site's bot protection refused the request. 4 of the 5 scraping APIs whose billing rule covers it do not charge for one, nor do Crawlbase, HasData, ScrapingAnt and ScrapeBadger, whose rules say a blocked request is free; Firecrawl charges it like any page it returns. Getting past it costs more than the 403 itself: the protected-site mode that fixes most 403s costs 3.4 times the plain rate on the middle one of 11 APIs, from $0.99 per 1,000 pages on Decodo Web Scraping API.
Each rule is taken from the vendor's own billing documentation as recorded on its page here; each rate is worked out from its own pricing page for a 100,000-page month.| API | Plain page, per 1,000 | Protected mode, per 1,000 | Mode | A 403 is |
|---|---|---|---|---|
| Decodo Web Scraping API | $0.49 | $0.99 | premium proxies | Not named |
| Scrape.do | $0.29 | $0.99 | residential proxies | Not charged (not on its list) |
| ScrapeOwl | $0.29 | $0.99 | residential proxies | Not named |
| WebScraping.AI | $0.20 | $0.99 | residential proxies | Not named |
| HasData | $0.59 | $1.19 | residential proxies | Not charged when blocked |
| ScrapeBadger | $0.49 | $1.29 | residential proxies | Not charged when blocked |
| ScraperAPI | $0.49 | $1.49 | premium proxies | Not charged (not on its list) |
| ZenRows | $0.39 | $1.99 | premium proxies | Not named |
| ScrapingAnt | $0.19 | $2.49 | residential proxies | Not charged when blocked |
| ScrapingBee | $0.49 | $5.99 | stealth mode | Not charged (not on its list) |
| ScrapingBot | $0.50 | $6.00 | stealth mode | Not charged |
Retrying a 403 in the same mode costs nothing where it is not charged, but usually fails again; the retry that works is the protected mode, billed at its own rate. 2 more APIs with a rule for it publish no separate protected-site rate: Firecrawl, which charges a 403 and Crawlbase, which does not charge one.
What a 403 means when you scrape
A 403 Forbidden is the site saying it understood the request and will not serve it. When a scraper gets one on a page that opens in a browser, the cause is almost always the site’s bot protection: Cloudflare, DataDome, Akamai or a similar service decided the request did not come from a person. The other causes are rarer: a page that needs a login, or a site that refuses visitors from some countries.
Why it happens
Bot protection scores each request on signals a plain HTTP client gets wrong: an IP address from a data centre, request headers in an order no browser sends, a TLS handshake that does not match the claimed browser, no JavaScript run, no cookies carried from the challenge page. One weak signal can be enough on a strict site. That is why a request that works on a small blog gets a 403 on a large shop.
How to stop getting it
Scraping APIs fix 403s by sending the request the way a browser would, from an address the site trusts. In order of cost:
- Rendering the page in a browser, which runs the site’s JavaScript checks and carries its cookies.
- Residential or premium proxies, addresses that belong to home or mobile connections rather than data centres.
- A stealth or protected-site mode, which combines both with browser fingerprinting and retries.
Try the cheapest step first on a sample of your pages; many sites stop returning 403s at the first or second step, and paying for the strongest mode on every page can cost more than ten times the plain rate. The table above gives each API’s protected-mode rate; the bot protection pages price each protection’s mode separately.
Who pays for the 403
Most APIs treat a 403 as their failure to deliver and do not charge for it, so a blocked request costs time, not credits. Firecrawl is the exception among the APIs that publish a rule: it charges for any page it returns, a 403 included. On APIs that charge the protected mode only when it succeeds, switching to it costs nothing on the pages that still fail.