Why was my scraping bill five times the estimate?

Your estimate probably counted accepted pages while the provider counted retries, a higher plan, target-specific credits, a different proxy pool, rendering, or time and bandwidth. A 5x bill does not require a price change: the worked Apify example below moves from $0.20 to $1.00 because five one-hour, 1 GB runs consume five compute units even when only the last run succeeds. (Apify pricing, Apify usage rules, retrieved 2026-07-26)

The fastest diagnosis is to stop comparing the invoice with your count of saved records. Compare it with attempted runs, the vendor’s per-response cost field, the exact target hostname and request options, and any session-time or bandwidth meters.

The six usual causes

The six usual causes table

1. Failed attempts and retries consumed billable resources

“Only pay for successful requests” is not a safe assumption across product classes. Firecrawl does not charge for Firecrawl-side failures such as timeouts or server errors, but it does charge when it fetches a page and the target returns a 4xx or 5xx response. A response that failed your job can therefore still consume the published 1-credit basic-page meter. (Firecrawl pricing, retrieved 2026-07-26)

Here is a fully worked failure example. The response count is hypothetical, but the failure rule and page meter are current published figures:

Workload in the logs:
4 fetched target errors rejected by your application
+ 1 fetched page accepted by your application

Estimate that counted only the accepted page:
1 fetched page x 1 credit = 1 credit

Vendor meter:
5 fetched pages x 1 credit = 5 credits

Difference:
5 credits / 1 credit = 5x

The example does not say Firecrawl charges every failed attempt. Firecrawl-side timeouts and server errors are excluded by its published rule. The five billed pages arise only when Firecrawl fetched all five and the target returned the four rejected 4xx or 5xx responses before the accepted page. (Firecrawl pricing, retrieved 2026-07-26)

Two other failure policies show why the ratio cannot be generalized. Scrapfly can disable failure protection when eligible failures exceed 30% over at least 1 hour, while listed Bright Data Custom Web Unlocker features bill 100% of requests. The blocked-scrape billing guide owns the full policy comparison. (Scrapfly billing and Bright Data Web Unlocker features, retrieved 2026-07-26)

Failure language can even conflict within one vendor. Browserless documentation and its pricing page disagree on whether the CAPTCHA meter applies to every attempt or only a successful solve. Do not resolve that conflict by guessing. The cloud-browser billing guide owns the full meter treatment; reconcile your logs against exported unit-consumption data or the invoice. (Browserless unit documentation, Browserless pricing, retrieved 2026-07-26)

2. A 550-request forecast miss became a plan-tier decision

The current WebScrapingAPI record supplies every vendor input used below: Starter costs $19 for 7,000 requests, Basic costs $49 for 20,000 requests, one successful response consumes one request, and neither plan has a published fixed overage rate. The planned and actual request counts are scenario inputs. (WebScrapingAPI pricing, successful-response rule, retrieved 2026-07-26)

Suppose the estimate allowed for 6,800 successful responses, but retries and extra URLs brought the billing-cycle total to 7,350:

forecast miss = 7,350 - 6,800 = 550 requests
Starter shortfall = 7,350 - 7,000 = 350 requests

Starter plan floor = $19
Basic plan floor = $49
plan step = $49 - $19 = $30
plan ratio = $49 / $19 = 2.578947x
unused Basic allowance = 20,000 - 7,350 = 12,650 requests

The $30 step, 2.578947 ratio, and 12,650-request remainder are calculated results. The stored vendor inputs are the two plan prices, the two allowances, and a null overage rate. The buyer may stop at 7,000 responses or move to another published plan, but the dataset cannot assign a made-up marginal price to response 7,001. (WebScrapingAPI pricing, successful-response rule, retrieved 2026-07-26)

3. The hostname changed the cost before your code did

ScraperAPI publishes a normal one-credit baseline and higher costs for named targets. The same API call shape can therefore consume a different number of credits when only the hostname changes. (ScraperAPI credit rules, retrieved 2026-07-26)

3. The hostname changed the cost before your code did table

The safe unit is therefore not “one URL.” It is “one successful request to this hostname with these parameters.” ScraperAPI returns the observed amount in sa-credit-cost and offers a URL-cost endpoint for checking the target and parameters before a run. (ScraperAPI credit rules, retrieved 2026-07-26)

4. Automatic routing changed the rate

Nimble publishes three replacement rates for successful Extract, Crawl, and Map URLs: VX6 standard without JavaScript at $0.0009 per URL, VX8 with JavaScript at $0.0013, and VX10 with JavaScript and stealth at $0.00145. Nimble says it selects the driver automatically and may escalate it. (Nimble pricing, retrieved 2026-07-26)

Assume a hypothetical 12,000-successful-URL workload was estimated entirely at VX6, but the billed mix contains 5,000 VX6 URLs and 7,000 VX10 URLs. The scenario mix is not a vendor success-rate or performance claim. It exists only to exercise the published rates. (Nimble pricing, retrieved 2026-07-26)

all-VX6 estimate = 12,000 x $0.0009 = $10.80

billed VX6 portion = 5,000 x $0.0009 = $4.50
billed VX10 portion = 7,000 x $0.00145 = $10.15
mixed-route bill = $4.50 + $10.15 = $14.65

difference = $14.65 - $10.80 = $3.85
bill ratio = $14.65 / $10.80 = 1.356481x

The $3.85 difference and 1.356481 ratio are calculated from the recorded per-URL rates and the stated mix. Preserve the final billed driver rather than labeling every URL with the rate assumed before dispatch. If the final driver is unavailable at request level, reconcile the vendor’s usage breakdown before attributing the difference. (Nimble pricing, retrieved 2026-07-26)

5. Rendering was enabled

ScrapeOwl’s current record assigns one credit to a basic request and five credits to a JavaScript-rendered request. These are replacement modes, not 1 + 5. For invoice diagnosis, compare the effective render setting with the setting assumed in the estimate. (ScrapeOwl request costs, retrieved 2026-07-26)

For a hypothetical 8,400-request workload, the published billing paths are:

basic mode = 8,400 x 1 credit = 8,400 credits
JavaScript mode = 8,400 x 5 credits = 42,000 credits
rendering multiplier = 5 / 1 = 5x

This is the exact five-times defect when the estimate assumed ScrapeOwl’s basic mode but the effective request used its JavaScript mode. Its residential and rendered-premium modes are separate 10-credit and 25-credit replacements, so this 5x result should not be applied to those modes. (ScrapeOwl request costs, retrieved 2026-07-26)

This is why “rendering is included” is not the same as “rendering has no meter.” Store the effective render setting in the log, including defaults applied by an SDK or workflow tool.

6. Time, bandwidth, and session minimums sat outside the request count

Browserbase’s $20 Developer plan includes 100 browser hours and 1 GB of proxy traffic. After those allowances, it publishes $0.12 per browser hour and $12 per GB. The cost guide also says every newly created browser session has a one-minute minimum billing period. (Browserbase pricing, Browserbase cost guide, retrieved 2026-07-26)

A request-only forecast cannot represent that bill. Two jobs with the same number of pages can use different browser hours, create different numbers of sessions, and transfer different amounts of proxy traffic. Keep session creation, session close, elapsed time, and proxy bytes next to the page-level result.

Bandwidth can also sit behind a request quota. ScrapeNinja includes 10,240 MB of monthly bandwidth on its paid plans and lists $0.001 per MB above that allowance, so the same request count can produce different overage when response sizes differ. (ScrapeNinja on RapidAPI, retrieved 2026-07-26)

When several causes stack

Do not multiply rates unless the vendor says to. ScrapingAnt publishes combined bundles, Scrapfly adds rendering to one selected proxy base, and ScrapeOwl publishes fixed replacement modes. The scraping-credit guide owns those combination rules; this guide needs only the diagnostic instruction to preserve the final request mode and returned cost. (ScrapingAnt credit table, Scrapfly billing, and ScrapeOwl request costs, retrieved 2026-07-26)

How to identify the cause from your own request logs

Log one row per attempt, not one row per saved record. Redact API keys and cookies, but keep:

How to identify the cause from your own request logs table

Capture the vendor’s own evidence when it is available:

How to identify the cause from your own request logs table

Then calculate:

observed billing units = sum(vendor cost field across every attempt)

accepted outputs = count(distinct item_id where your application accepted the result)

units per accepted output = observed billing units / accepted outputs

Split that ratio by target hostname, proxy mode, render mode, attempt number, and session ID. One comparison normally identifies the culprit:

• Cost on failed or superseded attempts points to failure billing or resource billing.
• A balance crossing the plan allowance points to the tier cliff.
• A jump grouped by hostname points to target-specific pricing.
• A jump grouped by effective proxy points to the proxy tier.
• A jump grouped by render or browser mode points to rendering.
• Growth with elapsed time, session count, or bytes points to secondary meters.

Finally, compare vendor success with application success. Firecrawl, for example, charges when a page was fetched but the target returned a 4xx or 5xx response, even though it does not charge for Firecrawl-side timeouts or server errors. A response can therefore be billable even when your application rejects it as missing or unusable. (Firecrawl pricing, retrieved 2026-07-26)

The next estimate

Build the next forecast from observed cost units per accepted output, not the cheapest advertised request. Test the actual target with the exact proxy and rendering parameters, preserve the returned cost field, include every retry and session, and show plan boundaries as decisions rather than inventing an overage rate.

Where the vendor does not publish a rule, keep it unknown. Browserless’s CAPTCHA wording conflicts across two current first-party pages, ScraperAPI does not publish a lower-tier per-credit overage, and dynamic target or protection profiles can change the effective configuration. Those are reasons to set a spending cap and collect request-level billing evidence, not reasons to fill the gap with an average.