How are you searching on the Web these days?
I still use Ecosia that realistically uses Google. Some, any way. You get many ads, a few links. Over time, though, the value of the links has continued to dwindle. When I want a deep search with follow up and deeper probe, I use Claude.ai.
But Claude does not show ads or links to sources (you can ask, but I usually don’t.)
Wonder what the ramifications are? Me, too. Then this article crossed my rss feed from Crazy, Stupid Tech.
Cloudflare’s CEO wants to save the web from AI’s oligarchs. Here’s why his plan isn’t crazy.
Written by Fred Vogelstein
Vogelstein writes:
Sixteen years ago Matthew Prince and classmate Michelle Zatlyn at Harvard Business School decided there was a better way to help companies handle hacker attacks to their websites. Prince and a friend had already built an open source system to help anyone with a website more easily track spammers. What if the three of them could leverage that into a company that not only tracked all internet threats but stopped them too?
Within months they had a business plan, won a prestigious Harvard Business School competition with it, and had seed funding. They unveiled the company, Cloudflare, a year later at the 2010 Techcrunch Disrupt completion, taking second place. And today, riding the explosion of cloud computing and armed with better technology and marketing, they’ve leapfrogged competitors to become one of the dominant cybersecurity/content delivery networks in the world.
I was immediately interested because I pay Cloudflare to protect my website. So far, it has been an excellent purchase.
Prince wants to talk about the future of the web and journalism with me because he thinks the AI chatbot revolution is killing both of them. And he thinks he can help fix that with something he calls pay-per-crawl, a gambit he and Cloudflare launched on July 1. He cares, he says, because “I love the smell of printer ink and a big wet press. So I kind of have a soft spot for the media industry and how important it is.” This isn’t spin. Two years ago he and his wife bought the Park City Record, his hometown local paper..
It’s a big enough problem that Prince and most publishers now believe a lot of journalism and anything else advertising supported online will die or be subsumed into AI companies in the next few years without intervention. And because 20 percent of websites use Cloudflare for security and traffic management, he can block enough AI chatbots crawlers – with his customers’ permission, of course – to at least force some big AI companies to the negotiating table.
I worked as an editor in trade magazine publishing for about 15 years. By the time I left in 2013, I could see the handwriting on the wall. During my entrepreneurial life, I’ve seen booms and busts. I can usually smell a bust coming. The advertising model of trade journals looked shaky. And digital funds are in short supply. In my 12 years as a blogger meeting many other former magazine writers also out on their own, I’ve met expenses with a little left over. But every year gets tougher.
The problem Prince is trying to solve is perhaps the biggest attack on how the web functions since Tim Berners-Lee created it in 1990. Until the AI revolution took hold two and a half years ago, the economic foundation worked like this: Search engines like Google and Microsoft freely and regularly indexed every site with web crawlers. In return, the search results powered by those indices generated referral traffic to those crawled websites. That has supported hundreds of billions of dollars in ad spending and search engine optimization.
But AI chatbots don’t work that way. Instead of ten blue links to choose from after a search – with advertising displayed at the top and right of results – AI chatbots just supply you with the answer. It’s a much better experience for users. Twenty years ago Google’s founders themselves in interviews with me and elsewhere talked about Google search only being an intermediate step toward creating an answer engine like this.
The problem this creates, however, is that there are no ads when AI chatbots give you the answer. If there are links to sources, users almost never click on them. AI chatbots also drive up publishers’ bandwidth costs because they crawl thousands of times a day. Wikipedia said back in April that these bots had raised their bandwidth costs 50 percent.
To larger entities than mine, this is a double hit. I hope Cloudflare can ignite a movement to bring some sort of sense to this market.
I highly recommend reading the entire interview.
