Where the data comes from
Four sources, all public, all fetched by our own crawler. No data provider, no scraping service, no purchased dataset.
The Chrome Web Store sitemap
The store publishes a sitemap and advertises it in its own robots.txt. It lists every
extension's ID and URL across 41 files, which is how we know what exists. It carries no
popularity or freshness information, so it tells us what to look at and nothing about what we will
find.
Extension listing pages
The main source. One fetch of an extension's store page gives its name, publisher, category, user
band, rating, version, size, last-updated date, and the complete manifest it ships with. Listing
pages are permitted by the store's robots.txt; the sub-pages that are not permitted
— reviews, support, privacy, related items — we do not fetch.
The Chrome update endpoint
The endpoint Chrome itself calls to fetch an extension package. We use it sparingly: to confirm a removal independently of the listing, and to verify that the package served matches the manifest on the page. We do not keep the packages.
Firefox add-ons
Mozilla's public API, used only to say whether an equivalent add-on exists for Firefox. We match conservatively — an exact name plus a matching publisher or homepage — and leave anything ambiguous unmatched, because a confident wrong match is worse than an honest blank.
How the crawler behaves
- It identifies itself honestly, with a user agent naming this site.
- It reads
robots.txtfor every source and refuses to fetch a disallowed path, enforced in code rather than left to whoever writes the next script. - It works within a daily request budget and pauses between requests.
- It stores every raw response, so any figure we publish can be traced to what the store actually returned on a given day.
The full audit, including the robots findings for each host and the dates they were checked, is
in the repository as docs/sources.md.