Web Scraping API Alternative

Two different products, often confused

General-purpose extraction tools are infrastructure: give them a URL, get back content, with the transport problems handled. They are deliberately domain-agnostic, which is their strength — the same tool works on a retailer, a directory or a news site. A commerce index is the opposite: narrow, opinionated, and useless outside commerce, but it answers questions the generic tool structurally cannot.

The part teams underestimate

Getting content back is the visible problem, so it gets the attention and the budget. It is also the part that a vendor can sell you outright. What remains after the content arrives is the work that does not finish: reconciling the same product across sellers that share no identifier, normalising attributes and units, deciding which of two disagreeing readings to believe, noticing when a parser has silently started returning a strikethrough reference price instead of the live one, and accumulating history that has value only in proportion to how long you have been collecting it.

Where the generic tool is the right answer

• Your targets are not commerce sites, or are spread across domains no index covers.
• You need the raw document, not fields — for archival, compliance or your own parsing.
• The task is one-off: a research question, a migration, a single audit.
• You have an existing identity and storage layer and genuinely only need transport.

Where it stops being the right answer

• You are maintaining more parsers than features, and each retailer redesign costs a sprint.
• You need to answer 'is this a good price' — which requires history you do not have.
• You need the same product across sellers, and are writing matching heuristics per category.
• You need to explain to someone else where a number came from and how confident you are in it.

On what we do

Commerce Census is the second kind of product, and it is worth being precise about that rather than blurring the line for marketing reasons. We publish an index of resolved products with an append-only observation history, and every value carries the channel it came from and a confidence weight. Where a seller offers a direct interface or a structured feed, that is what a value is worth most from — and the provenance ladder on every response says which it was, so you can decline to act on weak evidence rather than discovering later that you did.

The legal question

Whether a given collection method is permissible depends on jurisdiction, the site's terms, whether authentication was involved, what the data is used for and whether personal data is in scope. Those factors interact, and the case law is not static. Anyone answering this in a sentence on a marketing page — in either direction — is not giving you something you can rely on. It is a question for a lawyer who can see your specific facts.

What it actually costs to maintain your own extraction

The cost that surprises teams is not the tooling, it is the maintenance surface. Every source you extract from is a contract nobody signed and nobody maintains on your behalf: page structures change, endpoints move, field names shift, and none of it is announced. A parser that silently starts returning a struck-through reference price instead of the live one keeps producing plausible numbers indefinitely, which is worse than crashing.

• Every source is an independent maintenance stream, and they do not fail at convenient times.
• Silent parser drift is the dangerous failure. Validate values against plausible ranges, not just against parsing success.
• Per-address rate limits mean throughput scales with address supply rather than with code quality — we measured roughly 14 requests before universal refusal at one major retailer.
• Identity is a separate problem from extraction and is not solved by getting more pages.
• History has to be stored deliberately from day one, or you will have fetched a great deal and retained nothing.

A useful decision rule

Count the parsers you maintain against the features you shipped last quarter. When the first number exceeds the second, extraction has stopped being infrastructure for your product and has become your product — and at that point the question is whether building a commerce data platform is the business you intended to be in.

Questions

Common questions

Related