Rebuilding the Internet for Agents

On September 20, twelve days after Meta launched its Muse personal agent, Amazon shut it out of Amazon.com. Muse users trying to shop got a popup saying an unauthorized AI agent was violating Amazon’s Conditions of Use. Amazon’s stated reasons are worth reading closely: Meta never told Amazon Muse would be shopping its store, the agent didn’t identify itself while browsing, and Amazon had concerns about how Muse handled customer login credentials. Meanwhile Shopify went the other direction and is wiring Muse into Shop Pay checkout.

It’s easy to read this as a platform fight about who owns the transaction, and it is partly that (Amazon is also suing Perplexity over its Comet agent, and sells its own agentic buying through Alexa). But I think the more durable lesson is about plumbing. Every one of Amazon’s complaints maps to a piece of internet infrastructure that was built for a human sitting at a browser and has no good answer yet for software acting on that human’s behalf: who is this, who sent it, what is it allowed to do, and where is it actually coming from.

The merchants can already tell. Fullstory published a dashboard last month that spots Muse sessions behaviorally, no name tag required, because the pointer teleports from click to click and a pure agent session never highlights, copies or pastes. On one travel site Muse landed on flight results more often than on the homepage, because it had figured out the URLs and skipped the front door. Some brands are blocking at the CDN, some are welcoming it, and most haven’t decided. Agents are already transacting, sites can detect them heuristically, and nobody has agreed on how an agent should present itself.

My view is that the fix isn’t better ways for agents to pass as human. It’s infrastructure that lets them be recognized as accountable agents: identified, authorized, coming from somewhere with a reputation, and paying their way. That takes four layers, and most of them don’t exist yet in usable form.

The trust stack was built for people in browsers

When a request hits a large retail, travel or banking site today, it gets scored long before anyone looks at a user agent string. The site (or its CDN and bot-management vendor) checks the source IP against reputation feeds, looks up which ASN announces it and whether that ASN is a residential ISP, a mobile carrier or a hosting provider, geolocates the address to a city, fingerprints the browser and device, and compares all of that against the cookie and account history it already has. None of it was designed as an identity system, but stacked together it works as one, and it works because for twenty-five years the strongest signal on the web was simply that a real person’s traffic comes from a real person’s ISP, in roughly the place that person lives.

Browser-driving agents break most of those assumptions at once. An agent booking my dinner in Boston typically runs in a cloud region in Virginia or Oregon, egresses from a hosting ASN with a reputation shaped by every scraper and credential-stuffer that shared the range before it, geolocates to a data center rather than to me, and presents a headless browser fingerprint. To the risk engine it looks like exactly the traffic it was built to stop. The AgentID draft at the IETF makes the same point from the other side: without stable agent identity, rate limiting falls back to IP addresses, and agents rotate those trivially.

The industry’s current workaround is ugly. Agent and scraping companies that need to look human buy residential proxy capacity, much of which is sourced from SDKs embedded in free apps and browser extensions on consumer devices whose owners have only the vaguest idea they are relaying someone else’s traffic. That gets the IP to look residential, but it is the opposite of accountable: the address belongs to a stranger, the location is random, and when something goes wrong there is no one to call. Outside the sites that offer approved APIs or bot-access programs, the choice today is mostly between honest traffic that gets blocked and disguised traffic that gets through. That’s a bad equilibrium for everyone, including the sites.

Layer one: which agent is this?

The identity layer is the furthest along, and it’s moving faster than I expected. Cloudflare’s Web Bot Auth uses HTTP Message Signatures (RFC 9421): the agent operator publishes a public key set at a well-known URL, signs every outbound request, and any origin or CDN can verify the signature without a handshake or shared secret. Cloudflare launched a signed agents category with ChatGPT agent, Block’s Goose, Browserbase and Anchor Browser in the first cohort, and the underlying architecture draft is now working its way through the IETF.

The best argument for this I’ve seen recently came from my friend Dan Rowe, a Principal Engineer at Wayfair, who spent a week comparing notes with peers across retail and other industries. Everyone has the same graph: agent traffic up and to the right, with order volume roughly flat. An agent will crawl 10, 20 or 100 product pages at machine speed to hand a short list back to one user, so the merchant pays a lot more to serve roughly the same number of purchases. Dan’s view is that blocking isn’t a winning strategy, and that once agent requests are verifiable at the request level, sites can stop paying to render the giant header, navigation, footer and upsell modules for a reader who ignores all of them. His ask of the agent builders is simple: turn on Web Bot Auth. I’d put it slightly more bluntly. Verified identity is the precondition for every economic fix merchants want to make, from lighter pages to differentiated rate limits to charging for access.

Web Bot Auth answers who sent the request. It doesn’t say on whose behalf the agent is acting, or with what authority. That’s the next layer, and the early specs are getting interesting. The AgentID draft puts the agent’s owner, its capabilities and its full delegation chain into a signed token. The Personal Agent Protocol is the one I’d watch. Its 0.3 draft, still in progress, moved in a telling direction: rather than inventing its own token and sign-in machinery, it now rides on plain OAuth, recommends Web Bot Auth signatures so existing bot defenses can recognize the agent operator, demotes password-based sign-in to a transitional extension slated for removal, and hands carts and checkout to the Universal Commerce Protocol. In other words, the identity and delegation pieces are starting to compose with standards merchants already run. If Muse had shown up at Amazon speaking something like it, the conversation would at least have started from identity and consent rather than detection.

Layer two: on whose behalf, and allowed to do what?

Knowing that a request came from Meta’s agent tells the site who to blame, but not who the customer is. A shopping agent is a delegate: it acts for one person, with that person’s account, address and payment method, inside limits that person set. The credential question is where Amazon drew its sharpest line, and rightly so. An agent that collects a user’s password and replays it into a login form is indistinguishable from an account takeover, and every fraud team on earth is built to treat it as one.

The protocols are converging on the same answer: the user signs in on the company’s own page and grants the agent scoped, revocable access, rather than handing over credentials. For multi-agent workflows, where Agent A spawns B which spawns C, the delegation chain matters too, because otherwise the service receiving C’s request has no idea who originally authorized it. The goal is accountability that survives the hops: every request traces back to a verified person or organization, with scopes that only narrow as authority is passed along.

What these specs mostly leave out is the network. Over time, strong cryptographic identity should make some network signals matter less. But merchants’ fraud and risk systems are built on IP reputation, ASN type and geolocation, and they aren’t going to be retired in the next few years just because a new header shows up. In the meantime a signed, delegated agent still arrives from a data center three time zones away, and the risk engine still sees a hosting ASN.

Layer three: accountable network origins

This is the piece I think is most underappreciated, probably because I spent a good chunk of my Battery years in networking and edge infrastructure (Cumulus, Fastly and Catchpoint) and still think about the world in prefixes and ASNs. Agents need network origins that are accountable: address space whose reputation belongs to vetted agent traffic rather than to whoever abused the range last, announced by an operator that will answer the phone when something goes wrong. And when it’s useful, they need geographic affinity with the people they act for. If an agent is booking and buying for me, an address in my metro, announced by an ISP that actually serves it, gives every existing fraud and risk system a signal it already knows how to read. It’s the accountable version of what residential proxies fake today.

Location isn’t proof of identity, and it shouldn’t be a universal requirement. If I’m traveling, or the agent works for a company rather than a person, metro-matching may be the wrong answer. But location matters for a lot of what a delegate does. Prices, tax, inventory, shipping and content licensing are local. Fraud models weight a mismatch between the account’s home and the request’s origin heavily, and they should. A customer who normally shops from Boston and suddenly appears in an Oregon data center trips alarms, and today that’s exactly what my agent looks like. Offering locality as a capability removes a whole class of false positives without asking every merchant to rewrite their risk stack.

The company I’ve seen think hardest about this is IPXO, which runs one of the larger IP address marketplaces and leasing platforms. [Disclosure: I’m an investor in IPXO.] Their approach is to build an access network for agents with the ISPs that own the access rather than around them: agent traffic leaves from a partner ISP’s network, while IPXO takes on the address management, customer vetting and abuse handling the ISP would otherwise worry about. A handful of metros are live today across Europe and North America, with many more planned.

The question I keep coming back to is whether this ends up as infrastructure a few large agent operators build for themselves or as a shared, neutral layer. My bet is the latter, for the same reason CDNs became shared: the ISP relationships, the address space and the abuse operations are expensive to replicate metro by metro, and no merchant wants to whitelist fifty proprietary agent networks.

Layer four: someone has to pay for the page

Identity, delegation and network accountability make agent commerce technically possible. Economic alignment makes it sustainable, and this may be the most consequential layer of the four. Dan’s traffic graph is the merchant version of the problem: more pages served, roughly the same orders. Publishers have it worse. A human visit can carry ad impressions, upsell exposure and discovery value. An agent visit usually carries none of that, often costs more to serve, and on a content site may replace the human visit entirely, even when the agent eventually drives a purchase somewhere else. The deeper question is who captures the value of discovery once an agent, not a person, is doing the browsing.

Once requests are signed, attributable and coming from an accountable network, sites finally have the inputs to price them: serve a stripped-down page or a structured API to verified agents, rate-limit unverified ones harder, and charge per request where the content is worth paying for. Cloudflare’s pay-per-crawl work and a handful of paid-access gateways are early versions of this, and I expect per-request settlement between agent operators and publishers to become a real line item.

The nice property of getting the lower layers right is that merchants and publishers stop having to choose between blocking and tolerating. They can welcome agents on their own terms, which is what Shopify is effectively doing with Muse, and what Amazon would presumably do too if the agent showed up identified, scoped and paying its way.

What I’m looking for

Put the four layers together and you get a fairly clear picture of what the agent-ready internet needs: signed identity on every request, delegated and scoped authority that traces back to a real person, network origins that are accountable and local when it matters, and a settlement layer so the sites doing the serving get paid. Today each layer exists in pieces, built by different companies, and very little of it is stitched together.

Four technical layers don’t necessarily make four markets. Cloudflare and the other edge providers could bundle identity, policy and monetization. The big agent platforms could own delegation. Merchants could steer agents toward their own APIs. So the investment question isn’t which layer is interesting, it’s which layer becomes the control point for trust, policy and economics between agents and the rest of the internet. I don’t think that’s settled yet, which is exactly why it’s a good time to be looking.

That’s where I’m spending time at Encoded. I’m interested in founders building the verification and policy layer that sits in front of origins and turns agent identity into decisions (serve, shape, charge, block), in accountable agent egress and IP reputation, in delegation and credential handling that keeps passwords out of model context entirely, and in observability that shows merchants what agents are actually doing on their sites.

What I’m not interested in is another way to make agents better at pretending to be human. If Amazon, Wayfair and their peers are all watching the same graph go up and to the right, the honest path is going to get paved and the disguised one is going to get blocked. The opportunity is building the infrastructure that lets agents be recognized as agents.

If you’re building in any of these layers, or you run the infrastructure on the other side and have a view on what you’d need to say yes to agent traffic, I’d love to hear from you.

-ab

Other articles