--- title: "Stop Sniffing Bots" date: "2026-07-11T18:44:51+01:00" url: "https://invis.net/stop-sniffing-bots/" author: "invisnet" license: "CC BY-ND 4.0" license_url: "https://creativecommons.org/licenses/by-nd/4.0/" site: "invis.net" copyright: "Copyright 2024-2026 Charles Lecklider. All rights reserved." disclaimer: "Personal website. Opinions are my own." categories: - "Article" tags: - "AI" - "bot detection" - "browser sniffing" - "caching" - "Consolidated Content" - "content negotiation" - "publishing" - "user agents" - "web standards" --- # Stop Sniffing Bots I've updated the [Consolidated Content draft](https://datatracker.ietf.org/doc/html/draft-consolidated-content-01) to `-01`. The protocol itself hasn't changed — `-00` and `-01` describe exactly the same thing. What has changed are the notes and appendices, because in the six months since `-00`, people have started remaking one of the web's worst films. When I published the first draft in January, [Cloudflare](https://blog.cloudflare.com/crawlers-click-ai-bots-training/) and plenty of publishers were already complaining about bot traffic causing higher server load, bandwidth usage, and bills. There's a compliment in there if you look: the bots were visiting because publishers had the information people wanted. Where else should the bots be going? I've worked in publishing — I understand the numbers. But if an abstract comes wrapped in a page weighing more than a megabyte, the problem isn't really the bots. Often there isn't even a complete response worth caching. The HTML is a shell; JavaScript assembles much of the page in the browser, while advertising and personalisation ensure that one visit doesn't look like the next. A bot can cache the shell, but not the information it came for. So it fetches the page again. And again. And again. The publisher sees another expensive request and concludes that the bot is behaving unreasonably, but the site hasn't given it anything stable to reuse. The answer seemed obvious: give the client a smaller, stable representation designed to convey the actual information. Content negotiation lets the client ask for it, AI lets publishers create it. A consolidated Markdown representation replaces several requests, is cheap to serve, and can actually be cached. ## Time After Time Bot sniffing now serves two very different purposes. One is to block agents. If your long-term strategy is to block your users' preferred tools for reading, summarising, translating, searching, comparing, or otherwise using your content, good luck with that. The other is to help them, and that's far more interesting. If an agent sits between a publisher and its user, the publisher has a simple goal: make sure the agent understands the site properly. You don't want it scraping the navigation, guessing at the main content, missing the footnotes, folding in the sidebar, only to come up with the wrong answer. However, they’re trying to solve the right problem in the wrong way. Last month, TIME's chief operating officer, Mark Howard, [described how TIME now separates the traffic](https://digiday.com/media/how-time-and-others-are-rebuilding-parts-of-the-web-for-ai-agents/). Humans get the ordinary website. Approved bots get stripped-down Markdown pages containing the content and metadata. Unapproved bots are blocked. That is very nearly Consolidated Content. The difference is how the client gets there: in my spec, the client asks. At TIME, the server tries to identify the client and decides for it. The word “bot” makes guessing the client’s identity sound more reasonable than it is. We imagine discrete, independent things roaming the web for their own purposes. Sometimes, of course, that's exactly what they are. Crawlers exist. Scrapers exist. Training runs exist. Abuse exists. Maybe one day AGI will too. But when a user asks software to read something, summarise something, compare something, translate something, explain something, or find something, the software fetches things because the user asked it to. We already have a name for that: a **user agent**. `curl` is a user agent. Chrome is a user agent. A local assistant fetching a page because its user asked a question is a user agent. Their capabilities, risks, economics, and social consequences differ, but at the protocol boundary, there is no new category here. Bot sniffing isn't new, either. I saw the original. ## The Phantom of the Opera In October 2001, [MSN decided Opera users were using the wrong user agent](https://alistapart.com/article/msn/) and blocked browsers other than Internet Explorer. Opera was perfectly capable of displaying the content; MSN had looked at the user agent, decided it was the wrong one, and refused to serve the content. Opera users already had a workaround for sites like this: Opera could pretend to be Internet Explorer. Two years later, MSN had a bigger budget. This time it recognised Opera and [served it a stylesheet meant for an old version of Internet Explorer](https://www.wiumlie.no/2003/2/msn/): the page rendered incorrectly, and it looked like Opera was broken. Of course, there was absolutely nothing wrong with Opera — if it pretended to be Internet Explorer, the site worked. Opera responded with a special ["Bork" edition](https://press.opera.com/2003/02/14/opera-releases-bork-edition/) that displayed MSN in Swedish Chef (some say that improved the content — I couldn't possibly comment). The server identified the user agent, decided which version it should receive, and served the wrong one. Users didn't switch their favourite browser because one website behaved badly — their favourite browser was their favourite browser. If a site mattered enough, they might grudgingly open Internet Explorer for that one task, mutter something unprintable, and go straight back to the browser they actually liked; if it didn't, they went somewhere else. Compatibility failures stuck to the site, not the browser. So browsers adapted. If useful content depended on being recognised as the right user agent, browsers learned to identify as the right user agent. Servers learned to detect the pretending, and browsers learned to pretend harder. This is how we ended up with `User-Agent` strings full of historical claims, compatibility hints, lies, and ritual invocations of Mozilla for reasons nobody should still have to understand. The nineties produced some excellent films, but browser sniffing wasn't one of them. It never became a so-bad-it's-good cult classic. Nobody found hidden genius in it twenty years later. We burned every copy we could find, and hoped nobody would ever make a sequel. ## The Redundant Remake A quarter of a century later, people aren't making a sequel — they're remaking the original, shot for shot. Bot sniffing is just browser sniffing by another name. TIME's system is supplied by TollBit. TollBit's [current documentation](https://docs.tollbit.com/docs/other) tells publishers to match the `User-Agent` header against a list of known bots and redirect the matches. Its [FAQ](https://docs.tollbit.com/docs/bot-management) says it generally updates that list quarterly; publishers then update their edge configurations manually. Bots without a recognised smell are identified by fingerprinting and machine-learning tools. In its [published Fastly example](https://docs.tollbit.com/docs/fastly), the match is a regular expression. If better treatment depends on being recognised, everyone else has an incentive to wear a disguise. The `User-Agent` string starts growing again. Most of the people sniffing bots today, from the producers to the runners, don't know it's a remake. The few who do increasingly aren't on set: "senior" now means expensive, not "seen this before" — but that's a different film. The original spent years in production, building a huge and detailed set to give the client somewhere to say what it could do and what it would like. We still have that set. There is no need to spend another decade in production before we stop sniffing things. The client can ask for what it wants, the server can answer, and nobody has to guess or pretend — or smell. That's all the spec is: negotiation, not detection. --- Copyright 2024-2026 Charles Lecklider. All rights reserved. Personal website. Opinions are my own.