What does it cost to crawl a list of URLs at 10K, 100K, and 1M?

Among offers that can be priced from URL count alone, recurring free tiers make 10,000 simple-tier HTML URLs cost $0 at ScrapingAnt and Diffbot; at 100,000, Zyte is lowest at $13, and at 1,000,000, ScrapingDog is lowest at $90. The cheapest vendor changes at both steps: ScrapingAnt or Diffbot at 10K, Zyte at 100K, then ScrapingDog at 1M. ScrapingAnt, Diffbot, Zyte, and ScrapingDog (retrieved 2026-07-26).

What this comparison prices

Each URL is submitted once for a public static HTML response at the vendor’s basic or simplest target tier. The table excludes JavaScript rendering, premium or residential routing, screenshots, AI extraction, target-specific surcharges, client retries, taxes, and annual discounts. It uses monthly self-serve pricing and assumes every submitted URL produces the vendor’s billable outcome. Recurring monthly free tiers count; one-time signup credits and trials do not. One-time request packs count when they are the vendor’s standard self-serve purchase. That is the only baseline that can be applied consistently across the current records.

The dollar cells use the site’s bill engine where its normalized meter and plan are compatible. A † marks a source correction, a first-party conflict, or a purchase condition the engine cannot represent. “Not published” is not treated as zero.

Full cost table

The plan shown is the cheapest published way to cover that column’s volume. Throughput is the limit on the selected plans in 10K / 100K / 1M order. NP means the vendor does not publish a usable numeric concurrency limit.

Full cost table table

The marked cells need context:

• ScrapingAnt and Diffbot each publish a recurring 10,000-unit free tier. The bill engine now applies ScrapingAnt’s compatible request meter and returns $0 at 10K. Diffbot’s record does not publish a request.basic billing unit, so its sourced $0 table cell is not presented as an engine result; its free plan is also limited to 5 calls per minute. ScrapingAnt and Diffbot (reviewed 2026-07-29).

• Zyte’s current page publishes $0.13 per 1,000 simple HTTP responses on PAYG and $0.10 per 1,000 at the $100 commitment. The bill engine now preserves the explicit PAYG meter and plan-specific commitment rate, producing $1.30, $13, and $100 for the three columns rather than relabelling the fee-less plan as a quote. Zyte pricing (reviewed 2026-07-29).

• Bright Data’s engine result at 1M is $1,299.80, but it gets that number by applying the Web Unlocker request meter to a Web Scraper API record plan. The comparable Web Unlocker calculation is $499 + 617,000 x $0.0013 = $1,301.10. Bright Data pricing (retrieved 2026-07-26).

• Browserbase includes 1,000 Fetch calls on Developer and charges $1 per 1,000 afterward. That makes 10K cost $20 + 9 x $1 = $29, and 100K cost $20 + 99 x $1 = $119. At 1M, the Startup card says $1 per 1,000 over its included 10,000, while the comparison table says $0.50; those two published paths make the cheapest bill either $1,019 on Developer or $594 on Startup. There is no honest single number until Browserbase resolves the conflict. Browserbase pricing (retrieved 2026-07-26).

• Olostep’s engine chooses the $99 Standard plan at 10K because it does not model credit packs. The published $9 Starter plan includes 5,000 requests, and a 10,000-credit pack costs $20, so the purchasable total is $29. Olostep pricing (retrieved 2026-07-26).

• Oxylabs publishes target- and plan-specific rates plus plan-specific top-up caps. The engine now applies the “other sites” rate on each eligible plan, excludes Custom+ because its commitment is not published, and enforces the top-up caps. That produces $49 on Micro, $110 on Starter, and $999 on Business for 10K, 100K, and 1M simple successful URLs. Oxylabs pricing (reviewed 2026-07-29).

• Browse AI’s plan guide says Professional monthly top-ups range from $0.022 to $0.017 per credit, but its FAQ on the same page says monthly Professional top-ups cost $0.030. The 10K plan price is unambiguous; the larger bills are not. Browse AI plan guide (retrieved 2026-07-26).

Vendors that can fetch the URLs but do not publish enough for this table

These are not exclusions for being expensive. They are products whose public model needs an input that a URL count does not supply, or whose current rate is not public.

Vendors that can fetch the URLs but do not publish enough for this table table

Worked numbers: Olostep purchase paths at 10K, 100K, and 1M

Olostep’s dataset record exposes every input needed for this calculation. A standard scrape costs 1 credit. Starter costs $9 and includes 5,000 requests, Standard costs $99 and includes 200,000, and Scale costs $399 and includes 1,000,000. Starter’s overage note lists a 10,000-credit top-up for $20, valid for six months. The separate x402 path records a $0.01 basic-request meter. The one-time 500-request trial is excluded under this guide’s baseline. Olostep pricing and x402 docs (retrieved 2026-07-25).

At 10,000 URLs, Starter is 5,000 credits short. The recorded top-up covers that gap, but its pack size leaves another 5,000 credits available after this job. Olostep pricing (retrieved 2026-07-25).

10,000 URLs x 1 credit = 10,000 credits required
5,000 Starter credits + 10,000 top-up credits = 15,000 credits purchased
10,000 - 5,000 = 5,000-credit shortfall before the top-up
$9 Starter + $20 top-up = $29
15,000 - 10,000 = 5,000 credits left

The remaining top-up credits are not treated as lost: the record says the pack is valid for six months. The $29 is the cash required for this purchase path, not a claim that all 15,000 purchased credits are consumed by the job. Olostep pricing (retrieved 2026-07-25).

At 100,000 URLs, Standard covers the job without a top-up. Olostep pricing (retrieved 2026-07-25).

100,000 URLs x 1 credit = 100,000 credits required
Standard includes 200,000 credits
$99 Standard bill
200,000 - 100,000 = 100,000 credits left

The table therefore uses $99, rather than extending Starter with an unpublished universal overage rate. Olostep publishes top-ups but does not publish one rate that can safely be applied to every paid plan. Olostep pricing (retrieved 2026-07-25).

At 1,000,000 URLs, Scale lands exactly on its recorded allowance. Olostep pricing (retrieved 2026-07-25).

1,000,000 URLs x 1 credit = 1,000,000 credits required
Scale includes 1,000,000 credits
$399 Scale bill
1,000,000 - 1,000,000 = 0 credits left

No top-up arithmetic or assumed overage rate is needed at 1M. Olostep pricing (retrieved 2026-07-25).

The x402 meter provides a second fully recorded purchase path. Applying its $0.01 basic-request rate to the same URL counts produces the totals below. x402 docs (retrieved 2026-07-25).

10,000 x $0.01 = $100; $100 - $29 plan-and-pack bill = $71
100,000 x $0.01 = $1,000; $1,000 - $99 plan bill = $901
1,000,000 x $0.01 = $10,000; $10,000 - $399 plan bill = $9,601

Those differences are calculated comparisons, not extra vendor prices. The exact cash outlay per submitted URL in the three selected plan paths is likewise calculated: $29 / 10,000 = $0.0029, $99 / 100,000 = $0.00099, and $399 / 1,000,000 = $0.000399. Olostep pricing and x402 docs (retrieved 2026-07-25).

Olostep says it bills only successfully completed requests. This guide retains its stated comparison assumption that every submitted URL produces the billable outcome, and it does not assign a success rate. Olostep pricing (retrieved 2026-07-25).

One domain versus many domains

For most request-metered APIs, domain count alone does not change the baseline bill. Target class does. ScraperAPI publishes 1 credit for a standard page, 5 for Amazon, 25 for Google or Bing, and 30 for LinkedIn. A mixed-domain list must therefore be bucketed by the applicable target cost before its credits are totaled. ScraperAPI pricing and credit docs (retrieved 2026-07-26).

Three vendors make domain count operationally important:

• ScrapingFish publishes 25 simultaneous requests per target domain. One domain remains capped at 25; a list spread across independent domains can have more total work in flight without exceeding any one domain’s cap. This affects elapsed time, not its $0.002 successful-request price. ScrapingFish (retrieved 2026-07-26).

• Crawlbase assigns a domain-complexity tier. One-domain work stays in one tier, while a many-domain list can contain several prices. The numeric tiers were not publicly retrievable, so no blended bill belongs in the table. Crawlbase docs (retrieved 2026-07-26).

• Browse AI limits Personal to 5 websites and Professional to 10; pages and subdomains sharing one core domain count as one website. Its monthly add-on price is $5 per website on Personal and $3 on Professional, paid annually. Domain count can therefore add a separate website-slot charge even when page credits are unchanged. Browse AI plan guide (retrieved 2026-07-26).

Concurrency determines time, not just plan fit

If a plan allows C simultaneous requests and the average completed request occupies a slot for L seconds, the ideal lower bound is:

elapsed seconds >= URL count x L / C

The table does not invent L, so it does not publish crawl durations. It does show the cap that controls them. A $9 ScrapeOps plan has only 1 thread, while its $99 1M plan has 50; ScrapingDog moves from 5 on Lite to 50 on Standard. ScrapeOps pricing and ScrapingDog pricing (retrieved 2026-07-26).

Rate limits need a different lower bound. Decodo publishes 10, 25, and 50 requests per second across its three paid tiers, while ScrapeNinja publishes 100, 200, and 300 per minute across Pro, Ultra, and Mega. Those are throughput limits, not stated simultaneous-request limits. Decodo pricing and ScrapeNinja pricing (retrieved 2026-07-26).

Do not read “unlimited concurrency” as zero elapsed time. Bright Data publishes no numeric cap, but target latency, client capacity, retry policy, and site-side limits still exist; this guide does not assign numbers to them. Bright Data pricing (retrieved 2026-07-26).

Is a crawl endpoint cheaper than N scrape calls?

Not among the vendors whose current records publish both sides. A crawl endpoint may discover links and manage a job, but none of the comparable published formulas discounts the page work:

Is a crawl endpoint cheaper than N scrape calls? table

Use Crawl when discovery, same-domain traversal, collection assembly, or scheduling is worth the endpoint behavior. For a list you already possess, the current published math provides no crawl-endpoint discount.