Home / Cost tools / RAG pipeline budget
RAG pipeline budget: the first crawl, then keeping it fresh
20,000 pages refreshed weekly, fetching only the 10% that change, with JavaScript: $25.60 for the first month and $7.60 for each month after on WebScraping.AI, $109.20 in the first year, the lowest of 13 web scraping APIs that return markdown with a complete bill for both.
20,000 pages, a weekly refresh and 10% changing are an example to replace with your own figures: nobody publishes a typical corpus. Fetching only the changed pages needs the site to say which changed, through dated sitemap entries or a feed; without that, every refresh fetches every page.
The first year on WebScraping.AI
| API | First month | Each month after | First year | Plans | Checked |
|---|---|---|---|---|---|
| WebScraping.AI | $25.60 | $7.60 | $109.20 | Pay as you go | Oct 12, 2026 |
| Decodo Web Scraping API | $49.00 | $19.00 | $258.00 | $49 plan, then $19 plan | Oct 12, 2026 |
| ScrapeOps | $54.00 | $19.00 | $263.00 | 500,000 credits, then 100,000 credits | Oct 12, 2026 |
| ZenRows | $55.00 | $19.00 | $264.00 | Build B2, then Build B1 | Oct 12, 2026 |
| Scrape.do | $29.00 | $29.00 | $348.00 | Hobby | Oct 12, 2026 |
| Hyperbrowser | $30.00 | $30.00 | $360.00 | Startup | Oct 12, 2026 |
| Scrapfly | $30.00 | $30.00 | $360.00 | Discovery | Oct 12, 2026 |
| Oxylabs Web Scraper API | $100.00 | $30.00 | $430.00 | Basic, then Starter | Oct 12, 2026 |
| Firecrawl | $99.00 | $34.00 | $473.00 | Standard, then Hobby | Oct 12, 2026 |
| ScrapingDog | $40.00 | $40.00 | $480.00 | Lite | Oct 12, 2026 |
What moves the bill
Each line changes one of your assumptions and keeps the rest, then prices the cheapest first year again.
- Refresh daily instead of weekly: $638.00 in the first year, +$528.80 (cheapest becomes Decodo Web Scraping API).
- Fetch every page on each refresh: $955.20 in the first year, +$846.00.
- Twice as many pages change (20%): $200.60 in the first year, +$91.40.
- Without JavaScript: $18.00 in the first year, −$91.20.
- Twice as many pages: $223.20 in the first year, +$114.00.
How the pages add up
| Part | First month | Each month after |
|---|---|---|
| First full crawl | 20,000 | 0 |
| Changed pages on each refresh | 6,000 | 8,000 |
| Pages billed | 26,000 | 8,000 |
The first month fetches every page once and then runs the rest of that month's refreshes; every month after is the refreshes alone. A weekly refresh counts as four a month and a daily one as thirty. Each month is priced on the plan that fits that month's pages, so the first year assumes you move to a smaller plan after the first month; some vendors make a downgrade wait for the next renewal. Every page is counted as succeeding: what failures add depends on each vendor's rule, and the failed-requests tool prices it.
Only APIs whose own pages list markdown output are ranked here, because markdown is what most pipelines feed a model. On most of them markdown costs the same as HTML; the output formats page lists the ones that publish an extra charge for a format.
What this bill leaves out
This is the bill for fetching the pages. Splitting them into chunks, turning the chunks into embeddings and the model that answers from them are billed by your model provider, usually by the token, and they come on top. Count them from the pages above: the first month embeds the whole corpus, and each month after embeds only the pages you fetched again.