Nearly seven in ten South African news websites take no specific action to block AI crawlers from scraping their content, according to a new report launched today by the Media Leadership Think Tank (MLTT) at the University of Pretoria’s Gordon Institute of Business Science (Gibs), the Journalism Relay Project, and the International Fund for Public Interest Media (IFPIM).
The Protocol Gap: South Africa, the second in a three-country series following a report on Brazil, examined the robots.txt files of 263 South African news websites to determine how publishers are managing access by AI bots that scrape content to train large language models and power AI-generated search results and assistants.
The study found that while 74.1% of South African news sites have a robots.txt file, only 30.4% use it to specifically block at least one AI crawler as of April 2026, a modest rise from 29.7% recorded in December 2025. Where publishers do act, they block an average of six AI bots each, with crawlers from Common Crawl, ByteDance, Google, OpenAI, Amazon, and Anthropic among the most frequently restricted.
Robots.txt is a simple text file publishers can add to their website to signal which automated bots, including AI crawlers, are permitted to access their content. It remains one of the few tools publishers have to manage AI access, though it is not legally binding and depends on voluntary compliance by AI companies.
Robots.txt functions purely as an access signal and offers no immediate protection against crawlers that choose not to comply. However, robots.txt has been used as evidence in ongoing legal complaints against unauthorised scraping in Canada, the United States, and the United Kingdom.
South Africa's blocking rate is higher in absolute terms than in the other two countries in the CTRL+J research collaboration. The first report in the series, The Protocol Gap: Brazil, published in December 2025, found that just 7.2% of Brazilian news sites blocked at least one AI crawler. Indonesia's early figures stand at 13%, up sharply from 5.8%. South Africa's media sector is also considerably smaller than Brazil's or Indonesia's, and the report finds the headline figure masks a deeper imbalance: the capacity to restrict AI crawler access is concentrated among large, well-resourced publishers, while smaller, community, and vernacular outlets remain largely exposed to scraping.
The report argues that as generative AI companies grow increasingly dependent on extracting value from online content, publishers need more than awareness of the risk, they need the resources and technical capacity to act on it. In the absence of clear legal frameworks, robots.txt has become a de facto governance tool for managing AI access, though the findings suggest it remains a tool only some publishers can currently afford to use well.
The report situates its findings within a changing policy landscape. South Africa's Competition Commission concluded its Media and Digital Platforms Market Inquiry with remedies aimed at strengthening publisher controls, while the Copyright Amendment Bill was returned to Parliament for further amendment in June following a Constitutional Court ruling. Internationally, the report points to developments including the UK Competition and Markets Authority's new requirements on Google Search and Brazil's competition authority opening its own inquiry into Google's conduct.
Together with the Brazil report and the forthcoming Indonesia study, the series gives Global South publishers comparative evidence of where they stand relative to each other, not only relative to platforms based in the Global North, evidence intended to inform stronger policy and better decisions inside newsrooms.
The report was launched via an online webinar moderated by MLTT director Michael Markovitz, featuring Irene Jay Liu (IFPIM), Sérgio Spagnuolo (Journalism Relay Project), and Ompha Tshamano (MLTT), followed by a moderated discussion with Daryl Dingley representing the PSS publisher group and Sbu Ngalwa of Our City News and the South African National Editors' Forum (SANEF).
The Protocol Gap: South Africa is now available here. A third report examining Indonesia will complete the series later this year.
