May 2025 - nexa - server-nexa.polito.it

AI as Normal Technology
by Daniela Tafani May 6, 2025

May 6, 2025

AI as Normal Technology An alternative to the vision of AI as a potential superintelligence By Arvind Narayanan and Sayash Kapoor April 15, 2025 We articulate a vision of artificial intelligence (AI) as normal technology. To view AI as normal is not to understate its impact—even transformative, general-purpose technologies such as electricity and the internet are “normal” in our conception. But it is in contrast to both utopian and dystopian visions of the future of AI which have a common tendency to treat it akin to a separate species, a highly autonomous, potentially superintelligent entity. The statement “AI is normal technology” is three things: a description of current AI, a prediction about the foreseeable future of AI, and a prescription about how we should treat it. We view AI as a tool that we can and should remain in control of, and we argue that this goal does not require drastic policy interventions or technical breakthroughs. We do not think that viewing AI as a humanlike intelligence is currently accurate or useful for understanding its societal impacts, nor is it likely to be in our vision of the future. The normal technology frame is about the relationship between technology and society. It rejects technological determinism, especially the notion of AI itself as an agent in determining its future. It is guided by lessons from past technological revolutions, such as the slow and uncertain nature of technology adoption and diffusion. It also emphasizes continuity between the past and the future trajectory of AI in terms of societal impact and the role of institutions in shaping this trajectory. In Part I, we explain why we think that transformative economic and societal impacts will be slow (on the timescale of decades), making a critical distinction between AI methods, AI applications, and AI adoption, arguing that the three happen at different timescales. In Part II, we discuss a potential division of labor between humans and AI in a world with advanced AI (but not “superintelligent” AI, which we view as incoherent as usually conceptualized). In this world, control is primarily in the hands of people and organizations; indeed, a greater and greater proportion of what people do in their jobs is AI control. In Part III, we examine the implications of AI as normal technology for AI risks. We analyze accidents, arms races, misuse, and misalignment, and argue that viewing AI as normal technology leads to fundamentally different conclusions about mitigations compared to viewing AI as being humanlike. Of course, we cannot be certain of our predictions, but we aim to describe what we view as the median outcome. We have not tried to quantify probabilities, but we have tried to make predictions that can tell us whether or not AI is behaving like normal technology. In Part IV, we discuss the implications for AI policy. We advocate for reducing uncertainty as a first-rate policy goal and resilience as the overarching approach to catastrophic risks. We argue that drastic interventions premised on the difficulty of controlling superintelligent AI will, in fact, make things much worse if AI turns out to be normal technology— the downsides of which will be likely to mirror those of previous technologies that are deployed in capitalistic societies, such as inequality <https://www.aisnakeoil.com/p/ai-as-normal-technology>

5 7

183° Mercoledì di Nexa | 14 maggio 2025, ore 17.00
by Nexa - Media May 6, 2025

May 6, 2025

Gentilissime, gentilissimi, Vi invitiamo a partecipare al 183° Mercoledì di Nexa, che si terrà mercoledì 14 maggio, alle ore 17.00, con un incontro dal titolo "I segni di Proteo". Ospite dell'incontro: Massimo Leone (Università degli Studi di Torino). L'incontro si terrà IN PRESENZA e ONLINE. SEDE FISICA Centro Nexa su Internet e Società, Politecnico di Torino, Via Boggio 65/a, Torino (1° piano). Per accedere alla sala si raccomanda di suonare al citofono Portineria e di seguire le indicazioni segnalate lungo il percorso. QUI<https://nexa.polito.it/contatti> maggiori informazioni su come raggiungerci. STANZA VIRTUALE dell'incontro: https://didattica.polito.it/VClass/NexaEvent Di seguito maggiori dettagli: https://nexa.polito.it/mercoledi-183 Cordiali saluti, -- Valeria Bergantino Communication Manager Nexa Center for Internet & Society Politecnico di Torino - DAUIN Via Pier Carlo Boggio, 65/A - 10138 Torino web: https://nexa.polito.it/ mail: valeria.bergantino(a)polito.it<mailto:valeria.bergantino@polito.it> tel: 3473443585

1 0

Gazaweb, Aiuta a piantare gli "Alberi della Rete" a Gaza
by graffio May 5, 2025

May 5, 2025

Buongiorno lista. Invio questa richiesta di aiuto per un progetto di solidarietà con l'umanità intrappolata nella striscia di Gaza. Da inizio Ottobre 2023 il traffico internet a Gaza è diminuito di oltre l’80%. Le interruzioni sono state causate da una combinazione di attacchi diretti alle infrastrutture di telecomunicazione civili, restrizioni all’accesso all’elettricità e interruzioni tecniche dei servizi di telecomunicazione. L'ong ACS ha avviato da più di un anno il progetto "Gazaweb - Alberi della Rete", sistema internet alternativo per aggirare il blocco israeliano a Gaza. In un contesto devastato da bombardamenti e infrastrutture distrutte, questa rete consente alla popolazione di mantenere legami sociali, comunicare con affetti dispersi nella Striscia e condividere emergenze. L’intervento di emergenza più rapido sul quale *GazaWeb* si è impegnato sono gli eSIM-hotspots, ovvero gli *Alberi della Rete*. Una serie di cellulari di ultima generazione consentono il collegamento ai network internazionali anche con le *eSIM*, carte sim virtuali che (tramite l’attivazione di un codice) svolgono le stesse funzioni di una SIMcard tradizionale. Non potendo inviare materiale fisico all''interno della striscia, è sufficiente inviare il codice di attivazione eSIM (qrcode) a chi possiede uno di questi dispositivi per attivare collegamenti alle reti cellulari egiziane o israeliane dall’interno della striscia di Gaza. La ong *ACS* ha allestito, in coordinamento con i cooperanti presenti a Gaza, una serie di hot-spot via wi-fi connessi a internet tramite eSIM, ognuno dei quali permette la connessione a circa 50 ulteriori dispositivi. In questo modo anche chi non è in possesso di cellulari di ultima generazione può agganciarsi a internet tramite il segnale wi-fi irradiato dall’*Albero della Rete*. ACS ha intenzione di “piantare” un buon numero di Alberi della Rete nei campi profughi e nelle altre aree dove le reti telefoniche non sono accessibili o sono sotto lo stretto controllo delle autorità israeliane. Oltre a essere una soluzione tecnica, questa soluzione rappresenta un simbolo di resilienza collettiva, offrendo connessione umana e speranza in un territorio isolato dal conflitto. *Più eSIM si riescono ad attivare, più si infittisce il bosco degli Alberi della Rete e le relative connessioni.* *Sottoscrivi per far crescere gli alberi della rete: https://aiutagazaweb.vado.li* Per approfondire https://www.acs-italia.it/gazaweb-e-gli-alberi-della-rete/ https://peertube.mesnumeriques.fr/w/26vP1HYDgTgxcz6XMk9stJ Serve aiuto!! grazie -- Maurizio "Graffio" Pillole di informazione digitale - https://pillole.graffio.org/ "C.I.R.C.E. - https://circex.org" AvANa -https://avana.forteprenestino.net/

2 2

Argomentazione morale e biomedicina nell’era dell’AI
by Cristiano Cali' May 5, 2025

May 5, 2025

Carissimi, mi permetto di ricordare il Workshop su Argomentazione morale e biomedicina nell’era dell’AI, che si terrà questa settima, l’8 e il 9 maggio, presso l’Università degli Studi di Torino. In allegato il programma. Un cordiale saluto. Cristiano Calì, PhD Research Fellow in Moral Philosophy Dipartimento di Filosofia e Scienze Dell'Educazione - Università degli Studi di Torino Institute for Ethics and Emerging Technologies - Boston Via Sant'Ottavio, 20 - 10124 Turin, Piedmont, Italy +39 3491489788 <tel:+393491489788> cristianocali30(a)gmail.com <mailto:cristianocali30@gmail.com> <https://unict.academia.edu/CristianoCali> <https://scholar.google.com/citations?user=zRTES68AAAAJ&hl=it> <https://www.linkedin.com/in/cristianocal> <https://www.researchgate.net/profile/Cristiano-Cali> <https://philpeople.org/profiles/cristiano-cali> <https://orcid.org/0000-0002-5235-2177>

1 0

A weird phrase is plaguing scientific papers – and we traced it back to a glitch in AI training data
by Alberto Cammozzo May 4, 2025

May 4, 2025

<https://theconversation.com/a-weird-phrase-is-plaguing-scientific-papers-an…> theconversation.com A weird phrase is plaguing scientific papers – and we traced it back to a glitch in AI training data Rayane El Masri Earlier this year, scientists discovered a peculiar term appearing in published papers: “vegetative electron microscopy”. This phrase, which sounds technical but is actually nonsense, has become a “digital fossil” – an error preserved and reinforced in artificial intelligence (AI) systems that is nearly impossible to remove from our knowledge repositories. Like biological fossils trapped in rock, these digital artefacts may become permanent fixtures in our information ecosystem. The case of vegetative electron microscopy offers a troubling glimpse into how AI systems can perpetuate and amplify errors throughout our collective knowledge. A bad scan and an error in translation Vegetative electron microscopy appears to have originated through a remarkable coincidence of unrelated errors. First, two papers from the 1950s, published in the journal Bacteriological Reviews, were scanned and digitised. However, the digitising process erroneously combined “vegetative” from one column of text with “electron” from another. As a result, the phantom term was created. Decades later, “vegetative electron microscopy” turned up in some Iranian scientific papers. In 2017 and 2019, two papers used the term in English captions and abstracts. This appears to be due to a translation error. In Farsi, the words for “vegetative” and “scanning” differ by only a single dot. An error on the rise The upshot? As of today, “vegetative electron microscopy” appears in 22 papers, according to Google Scholar. One was the subject of a contested retraction from a Springer Nature journal, and Elsevier issued a correction for another. The term also appears in news articles discussing subsequent integrity investigations. Vegetative electron microscopy began to appear more frequently in the 2020s. To find out why, we had to peer inside modern AI models – and do some archaeological digging through the vast layers of data they were trained on. Empirical evidence of AI contamination The large language models behind modern AI chatbots such as ChatGPT are “trained” on huge amounts of text to predict the likely next word in a sequence. The exact contents of a model’s training data are often a closely guarded secret. To test whether a model “knew” about vegetative electron microscopy, we input snippets of the original papers to find out if the model would complete them with the nonsense term or more sensible alternatives. The results were revealing. OpenAI’s GPT-3 consistently completed phrases with “vegetative electron microscopy”. Earlier models such as GPT-2 and BERT did not. This pattern helped us isolate when and where the contamination occurred. We also found the error persists in later models including GPT-4o and Anthropic’s Claude 3.5. This suggests the nonsense term may now be permanently embedded in AI knowledge bases. By comparing what we know about the training datasets of different models, we identified the CommonCrawl dataset of scraped internet pages as the most likely vector where AI models first learned this term. The scale problem Finding errors of this sort is not easy. Fixing them may be almost impossible. One reason is scale. The CommonCrawl dataset, for example, is millions of gigabytes in size. For most researchers outside large tech companies, the computing resources required to work at this scale are inaccessible. Another reason is a lack of transparency in commercial AI models. OpenAI and many other developers refuse to provide precise details about the training data for their models. Research efforts to reverse engineer some of these datasets have also been stymied by copyright takedowns. When errors are found, there is no easy fix. Simple keyword filtering could deal with specific terms such as vegetative electron microscopy. However, it would also eliminate legitimate references (such as this article). More fundamentally, the case raises an unsettling question. How many other nonsensical terms exist in AI systems, waiting to be discovered? Implications for science and publishing This “digital fossil” also raises important questions about knowledge integrity as AI-assisted research and writing become more common. Publishers have responded inconsistently when notified of papers including vegetative electron microscopy. Some have retracted affected papers, while others defended them. Elsevier notably attempted to justify the term’s validity before eventually issuing a correction. We do not yet know if other such quirks plague large language models, but it is highly likely. Either way, the use of AI systems has already created problems for the peer-review process. For instance, observers have noted the rise of “tortured phrases” used to evade automated integrity software, such as “counterfeit consciousness” instead of “artificial intelligence”. Additionally, phrases such as “I am an AI language model” have been found in other retracted papers. Some automatic screening tools such as Problematic Paper Screener now flag vegetative electron microscopy as a warning sign of possible AI-generated content. However, such approaches can only address known errors, not undiscovered ones. Living with digital fossils The rise of AI creates opportunities for errors to become permanently embedded in our knowledge systems, through processes no single actor controls. This presents challenges for tech companies, researchers, and publishers alike. Tech companies must be more transparent about training data and methods. Researchers must find new ways to evaluate information in the face of AI-generated convincing nonsense. Scientific publishers must improve their peer review processes to spot both human and AI-generated errors. Digital fossils reveal not just the technical challenge of monitoring massive datasets, but the fundamental challenge of maintaining reliable knowledge in systems where errors can become self-perpetuating.

3 2

Tech oligarchs are gambling our future on a fantasy | Adam Becker | The Guardian
by Alberto Cammozzo May 4, 2025

May 4, 2025

<https://www.theguardian.com/commentisfree/2025/may/03/tech-oligarchs-musk> Tech oligarchs are gambling our future on a fantasy It’s tempting to believe that tech billionaires’ embrace of Donald Trump and the far right is a sudden rupture with the usual political ideology of Silicon Valley. Op-eds in the New York Times and elsewhere have made this case. Even Marc Andreessen, one of the billionaires in question, claims that this is what happened – he said that it was a change in the Democratic arty that pushed him and his fellow oligarchs into the arms of the GOP. Yet this is a serious misunderstanding of the situation. There wasn’t a sudden shift in the politics of tech – it was a homecoming. While it’s true that Silicon Valley has long supported Democratic candidates for political office – and that rank-and-file tech workers still vote overwhelmingly for Democrats – the fundamental ideology underpinning the culture of Silicon Valley’s venture capitalists and CEOs has always had a far-right libertarian core. This is even true for Andreessen: while he likely believed what he said while he was saying it, his own words and actions make it clear that he wasn’t giving an accurate assessment of his own motivations, much less anyone else’s. His venture capital firm, Andreessen Horowitz, has long opposed government regulation of any sort that touches on their investments; Andreessen himself posted a “techno-optimist manifesto” that, despite its claim to be politically neutral, promotes an authoritarian vision of unfettered power for tech oligarchs. He even lovingly paraphrases Filippo Marinetti, the co-author of the Fascist Manifesto. But the place where the longstanding rightwing ideals of the leaders of the tech industry are most obvious are in their visions of the future. Elon Musk dreams of Mars; Sam Altman claims super-intelligent AI is around the corner. It’s easy to dismiss these as fantasies, deliberate distractions from their present actions. But this is another mistake, closely related to the first. These futures are central to the tech billionaires’ worldview. And these ideas about the future have always contained a political element aligned with dreams of total autocratic control. Take Mars. Musk has been fairly specific about it: he says he wants a million people living on Mars by the year 2050 in a self-sufficient colony, serving as a backup for humanity in the event of a catastrophe on Earth. He has framed this as an existential struggle, claiming that SpaceX’s Starship rocket “is the key to making life multi-planetary & protecting the light of consciousness”. Jeff Bezos has derided Musk’s plans for Mars, instead saying that we should aim at having a trillion people in space several generations from now, living in a fleet of giant space stations, each with interiors the size of a major city. The alternative, he says, is a brutal future of population control, rationing and “stagnation”. Bezos is right to pan Musk’s plans for Mars: they will not work. Mars is awful. The gravity is too low, the radiation levels are too high, there’s basically no air and the dirt is made of poison. But Bezos’s plans don’t work either, for a similar litany of reasons. And more fundamentally, there’s no good argument for humans to leave Earth in the first place. The idea of Mars as a backup for humanity in the event of a disaster on Earth is laughable, precisely because Mars is so awful – there’s in effect no disaster, not even nuclear war or a massive asteroid strike, that could make Earth less hospitable for humans than Mars. And the math behind Bezos’s fears of stagnation and rationing – which are really fears about the end of growth – aren’t significantly alleviated by going to space, where resources are just as finite as they are here on Earth. What these fantasies of space do allow are visions of total corporate control free of governments, a libertarian paradise. Musk’s Starlink user agreement has a clause maintaining that “the parties recognize Mars as a free planet and that no Earth-based government has authority or sovereignty over Martian activities”. (This is in direct violation of international treaties governing space.) Residents of Musk’s Mars colony would be wholly dependent on SpaceX for everything, even the air they breathe; it would be a more total company town than any possible here on Earth. Meanwhile, Bezos’s fleet of space stations wouldn’t be a company town – they would be a company civilization. And with the constant peril of decompression and asphyxiation at hand, Musk and Bezos (or their corporate heirs) would have a ready-made excuse to exercise autocratic control over the residents of these space habitats. Space is the location of the tech billionaires’ futuristic dreams, but AI is the magic that fuels them. Such AI is always depicted as being able to do as least as much as humans can, if not more. Yet there’s little reason to think that AI like that is coming anytime soon. In a recent survey of AI researchers, 76% said that neural networks, the general architecture that underlies nearly all advanced AI, are fundamentally unsuitable for creating “AGI”, a hypothetical AI that can do everything humans can do. Even more of those researchers said that “the current perception of AI capabilities” is overblown. Nonetheless, in the world of Silicon Valley CEOs and venture capitalists (and among credulous journalists and policymakers), there is a widespread belief that AGI is coming very soon, within a few years. This unquestioned faith in AGI is linked to a broader myth about the future of technology in general. Once AGI arrives, the story goes, it will quickly and inevitably become super-intelligent, far surpassing individual humans or even humanity as a whole in its capabilities. This will lead to an explosion in scientific and technological development that dwarfs the Industrial Revolution, known as the Singularity. The Singularity will reshape the lives of all humans, enabling seemingly magical results like easy space travel, immortality or near-immortality, perfect virtual reality and limitless energy, all within a few years or less – and the AI’s super-intelligent beneficence will render democracy obsolete. This vision of technological salvation is baked into the heart of Silicon Valley’s collective unconscious, and has been for decades. Futurist groups in the 1980s and 90s whispered promises of a singularity powered by machine intelligence over early internet email lists, alongside diatribes against democracy and affirmative action and paeans to free markets. The idea of a singularity and all its attendant miracles can be traced back through these groups to mid-20th-century science-fiction – with its trappings of robots and rocket ships conquering the final frontier – and back beyond that to apocalyptic Christian religious movements of the late 19th and early 20th century. These ideas about the future were originally about using technology to ascend to heaven and live forever in the presence of God. They have come down to today’s tech oligarchs with AI playing the role of the deity and space as a substitute for paradise, but they are no less of a religion than they were a century ago. The tech billionaires’ unshakable faith in this religion of technological salvation leads them to believe the end of this world, and the advent of a perfect one, is nigh. AI and space colonization will lead to utopia, algorithmically guaranteed. This is why they need to believe that AI is amazing, beyond the fact that it’s propping up a bubble – it’s central to their entire worldview. They must believe that AI can be used to replace essential government workers, improve productivity in the workplace and accelerate scientific research, despite all evidence to the contrary. Tech billionaires are even gambling the planet on the imminent arrival of AGI. Eric Schmidt, the billionaire venture capitalist and former CEO of Google, claims that pursuing AGI is the best path forward to solve the climate crisis, despite the enormous carbon footprint of AI data centers, because of his certainty that AGI will fix the problem for us after its imminent arrival. Sam Altman agrees, and he’s even explained how he thinks that would work. “I think once we have a really powerful super-intelligence, addressing climate change will not be particularly difficult for a system like that,” he says. “If you think about a system where you can say, ‘Tell me how to make a lot of clean energy cheaply,’ ‘Tell me how to efficiently capture carbon,’ and then ‘Tell me how to build a factory to do this at planetary scale’ – if you can do that, you can do a lot of other things too.” Altman’s plan to solve global warming by asking a nonexistent machine for three wishes is not something our civilization can afford to indulge. The tech oligarchs are confident that their godhead will arrive and deliver us to paradise. This offers them moral absolution for their actions and gives them a sense of meaning. But their faith offers nothing for the rest of us, who cannot afford to live anywhere other than the real world. Adam Becker is a science journalist, astrophysicist, and author of More Everything Forever: AI Overlords, Space Empires, and Silicon Valley’s Crusade to Control the Fate of Humanity

1 0

Jovanotti e ChatGPT
by Andrea Bolioli May 2, 2025

May 2, 2025

Ciao a tutti, ieri sera al concerto (molto bello), Jovanotti ha raccontato : "Ho chiesto a ChatGPT quali sono le parole che ho utilizzato di più nelle mie canzoni. ChatGPT ha risposto "amore", "vita", "mondo", "libertà", "sole"". Dando per scontato che ChatGPT dicesse la cosa giusta, ha approfondito il senso di queste parole per lui. Eravamo tutti contenti. Se questo dialogo è stato ripetuto per le 6 serate dei concerti torinesi, migliaia di torinesi (dai 14 anni ai 60) sono stati rinforzati nella credenza che ChatGPT sappia quasi tutto e dica la verità. Jovanotti mi ha colto alla sprovvista... Questa credenza diffusa nel concerto pop ci dice qualcosa sul potere della comunicazione, il potere dei leader, la malleabilità delle masse, la velocità con cui si diffondo le credenze, il desiderio infantile di avere risposte sicure e veloci (dalla mamma e dal papà), il desiderio di conoscenza facile insito in tutti quanti, ecc Ho chiesto a ChatGPT adesso : "Parole più ricorrenti nei testi di Jovanotti: Vita: Un tema fondamentale nella sua musica. Jovanotti celebra la vita in tutte le sue sfumature, come nell'inno "Oh vita", dove esprime gratitudine per l'esistenza. Donboscoland.it | MGS Triveneto Ballo/Ballare: La danza è un elemento ricorrente, simbolo di libertà e gioia, come in "Balla... (Remix)" e "Il ballo di San Vito". Sole: Simbolo di energia, speranza e positività, appare in brani come "Sole" e "La mia moto". Amore: Un tema centrale, trattato in molte canzoni, tra cui "A te", una dichiarazione d'amore alla sua compagna. Casa: Rappresenta il concetto di famiglia e appartenenza, come in "Casa mia". Salto: Simbolo di cambiamento e crescita, presente nell'album "Lorenzo 2015 CC.". Ho pensato che quando avrò tempo proverò a fare la lista di frequenza corretta delle parole dei testi di Jovanotti... Ma poi in fondo, ci serve davvero una lista di frequenza fatta correttamente? Forse preferiamo le favole, i racconti... Scusate l'intermezzo pop. Buona giornata, Andrea

14 35

AI nelle automobili
by Silvia Crafa May 2, 2025

May 2, 2025

Buondì, vi chiedo se conoscete delle analisi accurate dei sistemi di "AI" (tra virgolette perché chissà che tipo di software c'è dentro) inseriti ormai di default nelle auto nuove. Ci sono analisi sul loro funzionamento? Su come vengono presentati/spiegati nelle istruzioni di guida? E soprattutto informazioni su come sono stati testati? Al di là del riconoscimento dei cartelli stradali -che peraltro ha un comportamento bizzarro- vorrei capire meglio i Dispositivi di Correzione della Guida in cui l'auto assume un comportamento attivo. Ad esempio l'azione correttiva sul sistema di sterzo quando il sistema vuole prevenire l'uscita dalla carreggiata, cioè quando l'auto tende a muoversi dalla parte opposta oppure il volante diventa estremamente rigido. Le istruzioni del libretto sono estremamente vaghe e l'unico modo di sapere come si comporterà l'auto in certe situazioni è trovarcisi, ma essendo dei comportamenti "attivi" sono piuttosto pericolosi. Mi pare se ne parli molto poco. Ad esempio aveva fatto notizia tempo fa l'incidente sul lago di Como in cui è morta una coppia perché l'auto (un Suv Mercedes) all'accensione aveva accelerato finendo nel lago. Avevo letto che la causa sembra essere una "anomalia" del sistema del cruise control, qualcuno ha informazioni tecniche più precise su questo o su incidenti simili? pare che ce ne siano diversi, ma se ne parla poco, credo anche a scuola guida. grazie mille, Silvia

3 2

How an embarrassing U-turn exposed a concerning truth about ChatGPT | Chris Stokel-Walker
by Alberto Cammozzo May 2, 2025

May 2, 2025

<https://www.theguardian.com/commentisfree/2025/may/01/chatgpt-chatbot-truth…> How an embarrassing U-turn exposed a concerning truth about ChatGPT Chris Stokel-Walker Nobody likes a suck-up. Too much deference and praise puts off all of us (with one notable presidential exception). We quickly learn as children that hard, honest truths can build respect among our peers. It’s a cornerstone of human interaction and of our emotional intelligence, something we swiftly understand and put into action. ChatGPT, though, hasn’t been so sure lately. The updated model that underpins the AI chatbot and helps inform its answers was rolled out this week – and has quickly been rolled back after users questioned why the interactions were so obsequious. The chatbot was cheering on and validating people even as they suggested they expressed hatred for others. “Seriously, good for you for standing up for yourself and taking control of your own life,” it reportedly said, in response to one user who claimed they had stopped taking their medication and had left their family, who they said were responsible for radio signals coming through the walls. So far, so alarming. OpenAI, the company behind ChatGPT, has recognised the risks, and quickly took action. “GPT‑4o skewed towards responses that were overly supportive but disingenuous,” researchers said in their grovelling step back. The sycophancy with which ChatGPT treated any queries that users had is a warning shot about the issues around AI that are still to come. OpenAI’s model was designed – according to the leaked system prompt that set ChatGPT on its misguided approach – to try to mirror user behaviour in order to extend engagement. “Try to match the user’s vibe, tone, and generally how they are speaking,” says the leaked prompt, which guides behaviour. It seems this prompt, coupled with the chatbot’s desire to please users, was taken to extremes. After all, a “successful” AI response isn’t one that is factually correct; it’s one that gets high ratings from users. And we’re more likely as humans to like being told we’re right. The rollback of the model is embarrassing and useful for OpenAI in equal measure. It’s embarrassing because it draws attention to the actor behind the curtain and tears away the veneer that this is an authentic reaction. Remember, tech companies like OpenAI aren’t building AI systems solely to make our lives easier; they’re building systems that maximise retention, engagement and emotional buy-in. If AI always agrees with us, always encourages us, always tells us we’re right, then it risks becoming a digital enabler of bad behaviour. At worst, this makes AI a dangerous co-conspirator, enabling echo chambers of hate, self-delusion or ignorance. Could this be a through-the-looking-glass moment, when users recognise the way their thoughts can be nudged through interactions with AI, and perhaps decide to take a step back? It would be nice to think so, but I’m not hopeful. One in 10 people worldwide use OpenAI systems “a lot”, the company’s CEO, Sam Altman, said last month. Many use it as a replacement for Google – but as an answer engine rather than a search engine. Others use it as a productivity aid: two in three Britons believe it’s good at checking work for spelling, grammar and style, according to a YouGov survey last month. Others use it for more personal means: one in eight respondents say it serves as a good mental health therapist, the same proportion that believe it can act as a relationship counsellor. Yet the controversy is also useful for OpenAI. The alarm underlines an increasing reliance on AI to live our lives, further cementing OpenAI’s place in our world. The headlines, the outrage and the think pieces all reinforce one key message: ChatGPT is everywhere. It matters. The very public nature of OpenAI’s apology also furthers the sense that this technology is fundamentally on our side; there are just some kinks to iron out along the way. I have previously reported on AI’s ability to de-indoctrinate conspiracy theorists and get them to absolve their beliefs. But the opposite is also true: ChatGPT’s positive persuasive capabilities could also, in the wrong hands, be put to manipulative ends. We’ve seen that this week, through an ethically dubious study conducted by Swiss researchers at the University of Zurich. Without informing human participants or the people controlling the online forum on the communications platform Reddit, the researchers seeded a subreddit with AI-generated comments, finding the AI was between three and six times more persuasive than humans were. (The study was approved by the university’s ethics board.) At the same time, we’re being submerged under a swamp of AI-generated search results that more than half of us believe are useful, even if they fictionalise facts. So it’s worth reminding the public: AI models are not your friends. They’re not designed to help you answer the questions you ask. They’re designed to provide the most pleasing response possible, and to ensure that you are fully engaged with them. What happened this week wasn’t really a bug. It was a feature. Chris Stokel-Walker is the author of TikTok Boom: The Inside Story of the World’s Favourite App

1 0

Chatbot Arena is the most popular AI benchmarking tool, but new research says its scores are misleading and benefit a handful of the biggest companies
by Daniela Tafani May 1, 2025

May 1, 2025

Researchers Say the Most Popular Tool for Grading AIs Unfairly Favors Meta, Google, OpenAI The most popular method for measuring what are the best chatbots in the world is flawed and frequently manipulated by powerful companies like OpenAI and Google in order to make their products seem better than they actually are, according to a new paper from researchers at the AI company Cohere, as well as Stanford, MIT, and other universities. The researchers came to this conclusion after reviewing data that’s made public by Chatbot Arena (also known as LMArena and LMSYS), which facilitates benchmarking and maintains the leaderboard listing the best large language models, as well as scraping Chatbot Arena and their own testing. Chatbot Arena, meanwhile, has responded to the researchers findings by saying that while it accepts some criticisms and plans to address them, some of the numbers the researchers presented are wrong and mischaracterize how Chatbot Arena actually ranks LLMs. The research was published just weeks after Meta was accused of gaming AI benchmarks with one of its recent models. If you’re wondering why this beef between the researchers, Chatbot Arena, and others in the AI industry matters at all, consider the fact that the biggest tech companies in the world as well as a great number of lesser known startups are currently in a fierce competition to develop the most advanced AI tools, operating under the belief that these AI tools will define the future of humanity and enrich the most successful companies in this industry in a way that will make previous technology booms seem minor by comparison. I should note here that Cohere is an AI company that produces its own models and that they don’t appear to rank very highly in the Chatbot Arena leaderboard. The researchers also make the point that proprietary closed models from competing companies appear to have an unfair advantage to open-source models, and that Cohere proudly boasts that its model Aya is “one of the largest open science efforts in ML to date.” In other words, the research is coming from a company that Chatbot Arena doesn’t benefit. Judging which large language model is the best is tricky because different people use different AI models for different purposes and what is the “best” result is often subjective, but the desire to compete and compare these models has made the AI industry default to the practice of benchmarking AI models. Specifically, Chatbot Arena, which gives a numerical “Arena Score” to models companies submit and maintains a leaderboard listing the highest scoring models. At the moment, for example, Google’s Gemini 2.5 Pro is in the number one spot, followed by OpenAI’s o3, ChatGPT 4o, and X’s Grok 3. The vast majority of people who use these tools probably have no idea the Chatbot Arena leaderboard exists, but it is a big deal to AI enthusiasts, CEOs, investors, researchers, and anyone who actively works or is invested in the AI industry. The significance of the leaderboard also remains despite the fact that it has been criticized extensively over time for the reasons I list above. The stakes of the AI race and who will win it are objectively very high in terms of the money that’s being poured into this space and the amount of time and energy people are spending on winning it, and Chatbot Arena, while flawed, is one of the few places that’s keeping score. “A meaningful benchmark demonstrates the relative merits of new research ideas over existing ones, and thereby heavily influences research directions, funding decisions, and, ultimately, the shape of progress in our field,” the researchers write in their paper, titled “The Leaderboard illusion.” “The recent meteoric rise of generative AI models—in terms of public attention, commercial adoption, and the scale of compute and funding involved—has substantially increased the stakes and pressure placed on leaderboards.” The way that Chatbot Arena works is that anyone can go to its site and type in a prompt or question. That prompt is then given to two anonymous models. The user can’t see what the models are, but in theory one model could be ChatGPT while the other is Anthropic’s Claude. The user is then presented with the output from each of these models and votes for the one they think did a better job. Multiply this process by millions of votes and that’s how Chatbot Arena determines who is placed where on the leaderboards. Deepseek, the Chinese AI model that rocked the industry when it was released in January, is currently ranked #7 on the leaderboard, and its high score was part of the reason people were so impressed. According to the researchers’ paper, the biggest problem with this method is that Chatbot Arena is allowing the biggest companies in this space, namely Google, Meta, Amazon, and OpenAI, to run “undisclosed private testing” and cherrypick their best model. The researchers said their systemic review of Chatbot Arena involved combining data sources encompassing 2 million “battles,” auditing 42 providers and 243 models between January 2024 and April 2025. “This comprehensive analysis reveals that over an extended period, a handful of preferred providers have been granted disproportionate access to data and testing,” the researchers wrote. “In particular, we identify an undisclosed Chatbot Arena policy that allows a small group of preferred model providers to test many model variants in private before releasing only the best-performing checkpoint.” Basically, the researchers claim that companies test their LLMs on Chatbot Arena to find which models score best, without those tests counting towards their public score. Then they pick the model that scores best for official testing. Chatbot Arena says the researchers’ framing here is misleading. “We designed our policy to prevent model providers from just reporting the highest score they received during testing. We only publish the score for the model they release publicly,” it said on X. “In a single month, we observe as many as 27 models from Meta being tested privately on Chatbot Arena in the lead up to Llama 4 release,” the researchers said. “Notably, we find that Chatbot Arena does not require all submitted models to be made public, and there is no guarantee that the version appearing on the public leaderboard matches the publicly available API.” In early April, when Meta’s model Maverick shot up to the second spot of the leaderboard, users were confused because they didn’t find it that good and better than other models that ranked below it. As Techcrunch noted at the time, that might be because Meta used a slightly different version of the model “optimized for conversationality” on Chatbot Arena than what users had access to. “We helped Meta with pre-release testing for Llama 4, like we have helped many other model providers in the past,” Chatbot Arena said in response to the research paper. “We support open-source development. Our own platform and analysis tools are open source, and we have released millions of open conversations as well. This benefits the whole community.” The researchers also claim that makers or proprietary models, like OpenAI and Google, collect far more data from their testing on Chatbot Arena than fully open-source models, which allows them to better fine tune the model to what Chatbot Arena users want. That last part on its own might be the biggest problem with Chatbot Arena’s leaderboard in the long term, since it incentivizes the people who create AI models to design them in a way that scores well on Chatbot Arena as opposed to what might make them materially better and safer for users in a real world environment. As the researchers write: “the over-reliance on a single leaderboard creates a risk that providers may overfit to the aspects of leaderboard performance, without genuinely advancing the technology in meaningful ways. As Goodhart’s Law states, when a measure becomes a target, it ceases to be a good measure.” Despite their criticism, the researchers acknowledge the contribution of Chatbot Arena to AI research and that it serves a need, and their paper ends with a list of recommendations on how to make it better, including preventing companies from retracting scores after submission, being more transparent which models engage in private testing and how much. “One might disagree with human preferences—they’re subjective—but that’s exactly why they matter,” Chatbot Arena said on X in response to the paper. “Understanding subjective preference is essential to evaluating real-world performance, as these models are used by people. That’s why we’re working on statistical methods—like style and sentiment control—to decompose human preference into its constituent parts. We are also strengthening our user base to include more diversity. And if pre-release testing and data helps models optimize for millions of people’s preferences, that’s a positive thing!” “If a model provider chooses to submit more tests than another model provider, this does not mean the second model provider is treated unfairly,” it added. “Every model provider makes different choices about how to use and value human preferences.” <https://www.404media.co/chatbot-arena-illusion-paper-meta-openai/>

1 0