
In November 2025, South Africa’s news industry secured something few markets have managed: a R688-million commitment from Google and YouTube, the outcome of the Competition Commission’s Media and Digital Platforms Market Inquiry.
The commitment followed the R300-million to R500-million a year the commission had estimated Google was extracting in value annually from South African publishers. Although the figures are not directly comparable, the agreement established a principle: platforms that profit from local journalism should contribute to its sustainability.
The next fight was left open. The commission found that AI chatbots had scraped South African news without agreement or compensation, and that AI companies have shown little interest in content deals with local publishers.
A September 2026 study of 263 South African news websites by the Media Leadership Think Tank at Gibs and partners found that only 30.4% use robots.txt, the file that tells crawlers which pages they may visit, to block even one AI crawler. That is barely up from 29.7% in December 2025. Meanwhile, leading local news sites lost about a fifth of their daily page views between May 2025 and May 2026, while AI chatbots referred less than 1% of their visits to news sites between January and May 2025, according to Competition Commission figures.
Publishers are negotiating half-blind
Cutting AI off entirely may not be the answer: AI search is becoming a new way to find journalism. What the industry needs is a fair price, and a fair price needs measurement. Here most newsrooms fall short. Tools such as Google Analytics track activity through the reader’s browser, but AI bots usually don’t run the JavaScript these tools rely on, so their visits go unrecorded. The reliable source is the server log: a record the website’s server creates for every request, human or not.
What the full picture looks like
Consider one client, a mid-sized European news publisher, between 5 September and 4 October 2026. The site served 58.9 million pages, and less than half went to humans. AI systems alone opened 4 million pages through 33 different AI agents, yet sent back just 7 418 traceable visits. For every 540 pages an AI read, one person clicked through. (The analysis was done with HoneyLog, a server-side log intelligence platform that shows bot and crawler visits versus human ones.)

Google’s crawler, by contrast, read 1.2 million pages and sent back nearly 1.5 million visits: the bargain publishers have lived with for 20 years. AI has turned it upside down, and not equally: for every reader it sent, OpenAI read about 78 pages, Perplexity about 117 and Anthropic’s Claude roughly 15 000.

South African newsrooms are very likely seeing the same, without a number to put on it.
Not all bots are equal
Treating “bots” as one group is how publishers end up blocking too much or too little.
- Search engine crawlers index pages and send readers back.
- AI training scrapers copy content to train models; nothing comes back. On the publisher above they made about 935 000 requests in a month: the strongest case for a licence fee, or a block.
- AI indexing crawlers (2.8 million requests) index content so a chatbot can answer questions, sometimes with a link to the source.
- AI citation agents fetch a page in real time because a person asked for it in a chat. Blocking them can mean vanishing from the answers people now read.
Then there are the impersonators: 10.4% of all requests came from bots posing as something else, such as fake Googlebots. Some 93% came from data centres, yet 83% were served pages as normal. Robots.txt means nothing to them.
From grievance to evidence
When AI companies come to the licensing table, they will bring their numbers. A publisher that can show, for each AI company, how many articles were taken, for what purpose and what came back is in a very different conversation from one that can only say “we think they’re scraping us”. Server logs are exactly the kind of usage data on which a licence can be priced, audited and renewed.
Measure first
The AI version of the R688-million argument won’t be won on principle, but by publishers who see 100% of their traffic and arrive with the numbers on the table.
You can’t price what you can’t see.
About HoneyLog
HoneyLog is a server-side analytics platform that reads server and content delivery network (CDN) logs to show newsrooms and online businesses every request hitting their site, from human readers to AI crawlers, scrapers and fake bots. It helps teams decide, with evidence, what to block, license or welcome. HoneyLog is part of the same group as Publytics. Follow HoneyLog on LinkedIn or visit honeylog.io.
- The author, Anna Battistella, is communications and marketing lead at HoneyLog and Publytics, two analytics companies that help publishers understand who and what is visiting their websites
- This promoted content was paid for by the party concerned





