How to Track Prices Across Retailers

The four decisions that determine whether it works

Almost every price tracking project succeeds or fails on the same four choices, and only the first is the one people spend their time on. Where the data comes from is genuinely the easy part. How you decide two listings are the same product, how often you re-read each one, how you store what you saw, and what you do when a source disagrees with itself — those are where a tracker either produces numbers you can act on or numbers that quietly stop being true.

Where the data actually comes from

There are four routes and they are not interchangeable. A direct API from the seller is the best data available and almost nobody offers one for competitive use. A merchant feed — the product file a seller supplies to a marketing channel — is authoritative but only covers sellers who publish one. A public product page carries the price a customer sees, which is the number that matters, but retrieving it at scale runs into edge filtering. A third-party provider resells one of the above and adds coverage breadth you would spend a year assembling.

Discovery is cheap. Prices are not.

This surprises most people building their first tracker. Finding out what products exist is nearly free — retailers publish machine-readable catalogue indexes deliberately, because they want their products found. We measured 11,160 product URLs from a single free request against one retailer's index, and about 2.5 million product IDs across 110 free requests at another. Knowing what exists costs almost nothing.

Prices are the opposite. Reading a current price means a request that renders like a customer's browser, and the cost is not per request — it is per address per time window. We measured roughly 14 successful requests from one address at one major retailer before every subsequent request from that address returned a refusal, including from a browser session that had been working seconds earlier. Recovery took over eight minutes. The unit of cost in price collection is address supply, not bandwidth.

Identity is the part that breaks silently

A tracker that fails to fetch tells you it failed. A tracker that matches the wrong products keeps running and keeps producing numbers, and nothing announces the problem. This is the failure mode that actually costs money.

Barcodes solve it where they exist, and they frequently do not. We measured barcode coverage varying from near-universal in packaged consumer goods to literally zero on public storefront endpoints, where the field exists in the schema and is empty on every store because the platform exposes it only through the merchant's own credentials. Where there is no barcode, matching falls back to brand, title and attributes, and the honest output is a match with a stated method and a confidence you can filter on — not a boolean.

• Same product, same seller — the trivial case, and still breaks when a seller relists under a new identifier.
• Same product, different sellers — needs a shared identifier or attribute matching with a confidence score.
• Different pack size, same product — arithmetically comparable only on unit price; comparing sticker prices across a 12oz and a 32oz is valid arithmetic and meaningless commerce.
• Successor model — last year's model at a discount is not the same product, and treating it as one makes every price trend look like deflation.
• Genuinely no equivalent — private-label goods often have no counterpart anywhere. The correct answer is that no comparison exists, not a loose match to something similar.

Cadence: how often is it worth re-reading?

A fixed schedule is wrong in both directions simultaneously. Most products do not change price for months, so re-reading them hourly spends your scarce address budget on confirming nothing happened. A small minority change several times a day, and reading those daily misses most of what they did.

The alternative is to drive cadence from each listing's own observed volatility, with a floor and a ceiling per source. The floor exists because no source should be read faster than it tolerates; the ceiling exists because a listing that has been flat for six months still needs an occasional check to confirm it still exists. Sources with generous limits sit near the floor permanently; sources with tight limits sit near their own, higher floor no matter how volatile the product is.

Store observations, never overwrite prices

The single most consequential storage decision: a price is not a field on a product, it is an observation with a timestamp and a source. If you store current price as a mutable column and update it, you have destroyed your history to save a row, and you will not be able to answer the only question that makes price tracking valuable — what was it before?

Append every observation. Derive current price as a view over the latest one. This also gives you provenance for free: each observation carries where it came from, and different sources deserve different confidence. A price read from a direct API is not the same evidence as a price parsed from a rendered page, and a system that flattens both into one number has thrown away the information needed to resolve a disagreement.

Computing history correctly is harder than it looks

Once you have observations, the obvious question is where today's price sits in the product's own history. The naive answer — count how many stored rows are below today's price — is wrong under any sane storage scheme, because you do not store one row per hour. You store a row when something changes. A price that held for sixty days is one row; a price that flickered for an afternoon is three. Counting rows says the afternoon mattered twenty times more than the two months.

The correct computation weights by the time each price was actually in effect. Reconstruct the intervals, weight by their duration, and compute the percentile as the share of time spent below today's price plus half the time spent at it. That last term matters more than it sounds: without it, a product whose price never moved reports as simultaneously at the 0th and 100th percentile depending on which comparison you write. With it, a flat price correctly reports as the middle of its own distribution — and the honest output for that product is not a percentile at all but a verdict that says the price has not moved.

How to evaluate a price tracking tool

• Ask when they started tracking your specific SKUs. History cannot be backfilled by anyone. A provider offering two years of depth on a product they began watching last month does not have it.
• Ask what identity method matched each pair, and whether you can filter results by match confidence. If matching is a boolean, you cannot triage it.
• Ask what happens when two sources disagree — whether the system surfaces the conflict or silently picks one.
• Test on your own SKUs, including the awkward ones: multipacks, refurbished units, private label, the item you know is listed inconsistently.
• Check whether historical figures are time-weighted or row-counted. Ask directly; the answer distinguishes a considered system from a database with a chart on top.

Build or buy

Build when you need a handful of sellers, control matters more than breadth, and you can start accumulating history now. The collection code for a narrow set is a weekend; the address budget and the ongoing maintenance as sources change are the real cost, and they are ongoing.

Buy when you need breadth across many sellers, when you need history that predates your decision to care, or when matching across a large heterogeneous catalogue is the actual problem — because that is the part that takes a year to get right and is invisible when it is wrong.

Questions

Common questions

Related