Ecommerce Product Data Collection

Two problems that get conflated

Discovery and pricing are different problems with different economics, and treating them as one is the most common architectural mistake in this area. Discovery asks what products exist and where. Pricing asks what each one currently costs. The first is nearly free and the second is not, which means they should run on separate schedules with separate budgets — and a system that re-discovers on every price cycle is paying repeatedly for an answer that changes slowly.

Discovery: measured, and cheaper than expected

Retailers publish machine-readable catalogue indexes on purpose. These are free, unauthenticated, uncapped in any practical sense, and they resolve to complete product URL lists. Measured across five major retailers: one index returned 11,160 product URLs in a single request; another resolved to roughly 2.5 million product IDs across 110 files at about 23,000 per file; a warehouse retailer's entire online catalogue came to roughly 8,070 products.

That last figure is worth sitting with. An entire national retailer's online assortment is a number most systems would treat as a single category. Catalogue size across the industry varies by four orders of magnitude, and a collection strategy tuned for a marketplace's hundreds of millions is badly wrong for a warehouse club's eight thousand.

Pricing: what a request actually returns

Because the constraint is address budget rather than requests, the number that determines whether a collection strategy is viable is priced products per request. We measured this across every source we could access:

A 1,534-to-1 spread between the best and worst source is not a tuning difference — it is the difference between a catalogue you can refresh daily and one you can refresh annually. Source selection dominates every other optimisation in this area.

What the payloads contain, and where they lie

A field being present in a schema does not mean it holds a value. We measured actual population rates rather than reading documentation, and the gaps are systematic rather than random.

• Barcodes: near-universal in packaged consumer goods, absent entirely from public storefront endpoints — the field exists on every product and is empty on all of them, because that platform exposes it only through the merchant's own credentials.
• Barcodes in search results vs detail pages: one retailer includes the barcode in its cheap bulk grid response; another omits it there and carries it only on the detail page. That single difference decides whether identity resolution is free or costs a second request per product.
• Reference price: public storefront payloads carry the seller's own compare-at price, which makes discount depth computable on a first observation with no history at all. Very few sources give you that.
• Availability: frequently a coarse in-stock flag rather than a quantity, and its refresh lag is often longer than the price's.
• Images: URLs are usually present, but some are session-bound and decay to a placeholder after the token expires — a failure that looks like working data until someone opens the page weeks later.

Rate limiting binds on addresses, not requests

The mental model that makes collection economics tractable: you are not buying requests, you are renting address-windows. We measured roughly 14 successful requests from a single address at one major retailer before every subsequent request from that address was refused — including a browser session that had been succeeding moments earlier, which confirms the limit is applied to the address and not to a client fingerprint. Recovery took more than eight minutes.

Once framed that way, the optimisation is obvious and the ranking changes: maximise items per request, spread reads across time rather than bursting, and prefer any source with an official route even when its per-request cap is lower, because an authenticated request does not consume the same scarce resource.

Edge behaviour is specific, not general

Sources differ in how they refuse, and the differences are reproducible. One retailer's edge silently hangs — no status code, no reset, just a read timeout — on any request whose User-Agent contains the conventional bot-contact format, the one well-behaved crawlers use to publish a contact URL. The identical request with a conventional agent string returned 200 in about 200 milliseconds. Others return a clean refusal status immediately, which is at least honest. Building against a general theory of how sites behave produces a system that fails in ways you cannot debug.

Storage: append, never overwrite

Every reading is an observation: a value, a time, a source and a confidence. Current state is a view over the latest observation, not a column you update. Overwriting is the one storage decision that cannot be reversed later — you can always derive current from history, and you can never derive history from current.

Questions

Common questions

Related