Don't block AI agents
Today, AI agents have to pretend to be human to the websites they use. This includes solving captchas, avoiding browser fingerprinting, and using residential IPs to avoid detection.
But why block AIs at all?
The “bot” users of old were generally unsophisticated and vampiric, existing to scrape, steal, and reproduce content. There’s no value is showing them ads, because their behaviour was pre-programmed and they didn’t have money to spend.
But this isn’t true for general-purpose web-browsing AI agents.
An AI agent’s behaviour is not fixed. Rather, they are able to react to the content on a website. Advanced agents are able to remember facts for next time, potentially longer than a human could. Adverts for AI may prove to be more effective than adverts for human users.
Less interesting questions to be asking:
- how do we invent harder captchas that humans can solve and AI can’t?
- how do we prevent non-human users from using our sites?
More interesting questions to be asking:
- why are we blocking bots at all?
- how can I optimise advertising for AI users?
- how can I otherwise monetise AI users?
The industry will be market-led: if it becomes far more profitable to serve AI agents than to block them, we’d expect permissive websites to win eventually.
But to find out, we have to give AI agents a chance.