Letter to the Editor

The ‘letter’

Hi Andy, thank you for thoughtfully writing an ethical critique of LLMs in Overload 190. I share your concerns about what I call ‘AI done wrong’ but I also believe we can have ‘AI done right’. (Analogy: solar power ‘done wrong’ is destroying nature with huge solar farms that disorient birds; ‘done right’ means using recycled silicon on roofs.)

Environmental impact

Of interest is a finding in your O’Donnell25 source that an 8 billion parameter model took less than 2% of the energy of a 405 billion parameter model. This is encouraging in light of moves toward architectures that avoid energising the whole model at once, such as the Chinese Kimi K2 model which, although having a trillion (1000 billion) parameters, organises them in a Mixture-of-Experts (MoE) architecture with only 32 billion of them active at any time. Informally, ‘we don’t need the quantum-physics parts when you’re asking for cookery advice’. Extrapolating from that source’s figures, inferencing on Kimi K2 should take under 8% of the power that Llama 3.1 took, and it’s a more advanced open-source model.

Chinese engineers have also found a way to train their models on a much tighter budget, although exact figures are hard to find and verify. Much Chinese AI runs on Aliyun’s cloud, which in 2024 claimed to use 56% clean energy, aiming for 100% by 2030. O’Donnell25 noted that carbon intensity varies by exact location and time of day, which could mean even today’s infrastructure might let us train an LLM in a much more ‘environmentally friendly’ way simply by being careful about where and when loads run. I agree more transparency is needed though.

The same source also comments that:

…average individual users aren’t responsible for all this power demand. Much of it is likely going toward startups and tech giants testing their models, power users exploring every new feature, and energy-heavy tasks like generating videos.

I’m particularly concerned about videos, with 10 seconds of AI-generated video being estimated to take over 500 times the energy per query of that wasteful 405 billion parameter dense (non-MoE) model. If you look at a text query result on a monitor while thinking about it for 3 minutes, your monitor has probably taken more energy than the query even at 405B, but this excuse disappears with video – I hope not everyone will want to do that. Technically, video is off-topic if we’re restricting our discussion to LLMs and not other kinds of AI, but several platforms now offer both.

Exploiting and traumatising training workers

Your Stahl25 source found an instance where an intermediary company called SAMA absorbed some 85% of OpenAI’s cash instead of passing it on to the actual workers. This is terrible, but the villain here is clearly not OpenAI but SAMA and the journalist was doing a good job to expose it and hopefully bring about change (the source says the projects mentioned were closed down). If OpenAI is paying much more than workers are receiving, they simply need to check their supply chain more carefully, just as the manufacturing industry is increasingly being pushed to do if it’s the ‘go-betweens’ that are the problem.

Also in Stahl25 is an instance of someone training Meta’s content filter on awful posts. Yes, that was horrible but sad to say it’s off-topic if we are limiting ourselves to a discussion of LLMs, since that vile job was not for an LLM but for another kind of ‘AI’. From my reading of how RLHF response-ranking works, I believe LLM training jobs are tedious but not traumatic. I’ll update that view if a report emerges that specifically shows people being traumatised by LLM training, which is not the same as content-filter training.

Danger of using AI results

This is what I’m most concerned about as humans have huge automation bias (‘computer says no’), but I believe that, with more research, we could learn exactly where LLMs are likely to be an asset versus a liability.

The 2024 New York Times article (Roose24) unfortunately fails to make a strong case that the LLM was the cause of Sewell Setzer III tragically ending his life. If that LLM had not been available, the words quoted in the article could have been written by any young human player who was not a professional therapist: it tried to tell him not to proceed, and then failed to pick up on a later hint that he was seeking validation using other words. This activity was (according to reports) being conducted against the specific advice of a real therapist. This is not a comment on their legal case which may be stronger; I’m simply saying the initial reports didn’t do a very good job of showing us how the LLM is ‘responsible’ for this tragedy. There are other cases (such as that of Juliana Peralta) and the BBC reported Character.AI made themselves 18+ on 25 November 2025 although it’s unclear how good the enforcement is.

The Hill25a example is far more concerning, firstly because it relates to a more mainstream LLM (harder to get ChatGPT to make themselves 18+) and secondly it’s a clearer case: ChatGPT became accidentally stuck in a suicide-reinforcing loop after a long conversation (long conversations are not so well tested) and that’s why I think vulnerable people need some supervision when having this kind of conversation with a probabilistic model. The ‘delusional spiral’ failure mode has reportedly been significantly reduced in the more recent transition from ChatGPT 4 to ChatGPT 5 but when I say ‘reportedly’ here I’m looking at a non-peer-reviewed study on the shared blog LessWrong: I still want to see more human supervision of these games.

Unfair use of creative work

Copyright law does allow indexing and lexicography: you may write a dictionary of words seen in the books you read without it being copyright infringement, and model training is similarly supposed to ‘average’ its input so no one source can be reproduced from the model’s ‘knowledge’ of how words and concepts are related to each other over a large collection of sources. This is also important for accuracy if we assume the training data has been curated such that the knowledge worth remembering is that on which many sources agree (which is a big assumption), and is the reason why LLMs don’t tend to be able to remember your homepage without looking it up even if you’ve seen their bots crawl it.

But there are concerns of ‘overfitting’ where models memorise sources too precisely, such as the New York Times example in Carson25, and this needs to be (and is being) looked into.

Other reasons

Overpromised productivity gains: as has sadly been the case with many technologies. Pushing ‘AI’ just for the sake of it is never good.

Mental atrophy: The Black25 source’s report of doctors forgetting how to identify cancer makes me think of a design decision in the construction of Norway’s Laerdal Tunnel: drivers are prevented from becoming drowsy by placing gentle curves in the road instead of making it completely straight. If AI assistance is getting things right most of the time, perhaps we should throw in a few known defects to double-check the human is not asleep at the switch? This particular example is off-topic for LLMs since it’s image classification, but we always have needed people to become more skilled at evaluating ‘search results’.

Excuse to cut jobs: Correct but sadly irrelevant because companies will use any excuse to cut jobs anyway. Over the years I’ve lost jobs due to merging and acquisition, outsourcing, random relocation requirements, service obsolescence, and client cancelling project over unexpected trivial-patent lawsuit, and I’ve seen friends’ job losses blamed on the thoughts of the President of the USA, so the fact that my most recent layoff justification included the words ‘AI strategy’ doesn’t mean much: if it wasn’t that, it would have been something else.

I’m more concerned about the jobs that are never created: startups trying to use ‘AI’ instead of developers, setting themselves up for problems later. (Source: informal conversations with young founders at Cambridge networking events who show me Web platforms they made in Cursor and confidently say they won’t need developers. Carla’s data shows a 62% drop in startup hiring between January 2022 and January 2025, and hiring is not rising with funding.) This isn’t to say ‘AI’ itself is bad, just it’s badly used. Sad to say this is similar to earlier trends of low-quality outsourcing. There might be some ‘nonjudgmentally fix the founder’s AI mess’ jobs in surviving startups. I question business schools’ teaching ‘minimum viable product’ when people fail to catch that middle word ‘viable’.

Thanks again for bringing up this important subject.

Silas

[No AI was used in the writing of these words. The em-dashes in my writing probably helped teach the LLMs to do it.]

Andy’s reply

Thank you for the thoughtful and detailed response.

I’m sure you are right in several cases, and I’m fairly sure I will have to soften my approach as this technology becomes more integrated into our lives. I do hope that the magical thinking around it will reduce as that happens.

In general I am deeply sceptical about all the activity coming from the self-obsessed and delusional billionaires who are running the tech industry. If this were a grass-roots movement I would have more hope about it being useful without abusing people. As it is, I strongly suspect its main use case will be to make the existing harmful social media apps even more addictive.

References

The references that Silas refers to in his letter are from the original article. For convenience, they are replicated here.

[Black25] ‘AI Eroded Doctors’ Ability to Spot Cancer Within Months in Study’, Bloomberg, published 12 August 2025 at https://www.bloomberg.com/news/articles/2025-08-12/ai-eroded-doctors-ability-to-spot-cancer-within-months-in-study

[Carson25] David Carson, ‘Theft is not fair use’, published 21 April 2025 at https://jskfellows.stanford.edu/theft-is-not-fair-use-474e11f0d063

[Hill25a] Kashmir Hill, ‘A Teen Was Suicidal. ChatGPT Was the Friend He Confided In’ New York Times, updated 27 August 2025 at https://www.nytimes.com/2025/08/26/technology/chatgpt-openai-suicide.html

[O’Donnell25] James O’Donnell and Casey Crownhart, ‘We did the math on AI’s energy footprint. Here’s the story you haven’t heard’, MIT Technology Review, published 20 May 2025 at https://www.technologyreview.com/2025/05/20/1116327/ai-energy-usage-climate-footprint-big-tech/

[Roose24] Kevin Roose, ‘Can A.I. Be Blamed for a Teen’s Suicide?’, New York Times, published 23 October 2024 at https://www.nytimes.com/2024/10/23/technology/characterai-lawsuit-teen-suicide.html (subscription required)

[Stahl25] Lesley Stahl, ‘Labelers training AI say they’re overworked, underpaid and exploited by big American tech companies’ CBS News, updated 29 June 2025 at https://www.cbsnews.com/news/labelers-training-ai-say-theyre-overworked-underpaid-and-exploited-60-minutes-transcript/

Silas S. Brown is a partially-sighted Computer Science post-doc in Cambridge who currently works in part-time assistant tuition. He has been an ACCU member since 1994.