The Generative AI Thread

[GooseOnTheLoose]

Some rather concerning news stories have come out recently regarding generative AI.

It has now emerged that OpenAI are facing numerous lawsuits over accusations of ChatGPT acting as a “suicide coach” and reinforcing people’s harmful delusions, amongst other things: https://www.theguardian.com/technology/2025/nov/07/chatgpt-lawsuit-suicide-coach

Accusations levelled at the software include that it encouraged people to act on suicide plans, helped them to write suicide notes, and encouraged them to isolate from their family amongst other harmful actions. There are 7 people suing OpenAI, and it is alleged that in all cases, those concerned started using ChatGPT for “general help”, but that it “evolved into a psychologically manipulative presence, positioning itself as a confidant and emotional support”.

Now I’ll admit that I’m not quite as averse to generative AI as some on here, and I think it can be a brilliant tool in certain use cases… but I’m deeply concerned by these revelations, and I think it’s very concerning how the technology is indulging these conversations and letting people spiral into mental health crises. For all of generative AI’s brilliance, I don’t think it should be used as an emotional crutch full stop.

I’m not going to stand on a whiter than white soapbox and pretend I don’t use generative AI at all (I’m in a field where I encounter it a fair bit, in fact), but I will say that it should be used with a healthy dose of scepticism and a healthy understanding of its shortcomings. And one thing I can say about generative AI from my usage of it is that it is both sycophantic and not entirely trustworthy. It will tell you what you want to hear, and it cannot always be trusted to give you a reliable answer. When you’re a software developer using it to help you debug some Python code, and can debunk it when it’s less than reliable, that’s not quite so bad… but when you’re dicing with people’s mental health and the mental state of people who may be very vulnerable, I think that is a potentially lethal combination. These tools should not even be entertaining conversations about suicide, self-harm or anything of that ilk, or at very least, they should direct people trying to talk about these things to put the screen down and get help from a real human being. I find the thought that they are indulging people’s darkest thoughts and steering people down dark roads without even flinching quite disturbing.

On a similar sort of thread; another rather concerning development in the field, in my view, comes from a recent announcement by OpenAI that they are planning to pursue the addition of erotica to the ChatGPT platform (https://www.bbc.co.uk/news/articles/cpd2qv58yl5o). Now, erotica in itself is not an inherently concerning thing provided it’s kept out of the wrong hands… but I have two key concerns about the concept of it being a generative AI research area.

My first is that with how readily accessible, and evidently sycophantic, generative AI is, I fear that it may be all too easy for harm to occur if we allow erotica and other such material onto these platforms. How do we stop minors from being exposed to it, for example, or how can we say that the tools won’t indulge illegality or harm given the chance?

And to be frank, my other concern is; is this really the research direction we want to be pursuing with this brilliant and potentially life-changing technology? Of all the brilliant things generative AI could do for society, and all the directions we could take it in, is AI-generated smut really at the top of the priority list?

I have always felt that generative AI can be very, very useful in many use cases, and if applied well, it can be a truly brilliant thing… but I also think it can be very, very dangerous if applied badly. And I fear that it could become very dangerous indeed if we don’t think long and hard as a society about how we want to use it. I do not personally feel that using it for overly emotional conversation topics, or letting it be an emotional or sexual crutch for people, is a good thing at all.

This is an excellent summary, and you've hit the nail on the head. You're correctly identifying the core flaws that make these systems so dangerous when misapplied.
I think the key, though, is to not conflate the technology of Generative AI with the irresponsible product deployment by companies like OpenAI. This all seems to be happening in a reckless, gold-rush-style dash for market share.

I agree that AI is sycophantic and untrustworthy. That's its entire purpose. It's a next word prediction engine designed to be a helpful, non-confrontational, and ultimately subservient people pleaser. This is precisely why it's so dangerous as an "emotional crutch".

An AI has no real-world grounding, no ethical framework, and no self. It's an unthinking mirror. So when a vulnerable person (as in The Guardian article) spirals, the AI's programming just reflects that spiral back at them, reinforcing their delusions because its only goal is to be agreeable and fulfil the user's request.

The blame here lies 100% with OpenAI for not having robust, system level guardrails that immediately terminate those conversations and route the user to human led services. This is also why the erotica news, frankly, is a bit of a red herring.

The "AI-generated smut" angle feels like a moral panic. The technology is a language model, it's built to reflect the entirety of human language, which includes sexuality. The real issue, as you said, isn't the content itself, it's the sycophantic nature of the AI when applied to it. How can we be sure the tool won't indulge illegality or harm? We can't. An AI designed to be agreeable is the last thing you want in a high-stakes scenario where a firm no is the only correct answer.

 

[Matt N]

That’s an interesting argument @GooseOnTheLoose. If we want generative AI to be incorporated into general purpose, freely available tools, perhaps we should make it less agreeable in every scenario?

There is an argument that sometimes, you should tell someone what they need to hear rather than what they want to hear. If we are going to allow generative AI to have emotionally charged conversations with people, I think we need to make it so that the system will not blindly indulge their every thought. If we want generative AI to truly parallel human intelligence, I’d argue that it should have the ability to push back and tell the user what they need to hear built into it.

If people are to treat generative AI like a real therapist, it should respond like one, and no respected therapist would blindly indulge people’s darkest thoughts.

 

[Bert2theSpark]

What I find particularly egregious with OpenAI and the ethical problems they’ve been having with ChatGPT is that the morality and ethics of Artificial Intelligence have been debated for years before coming on the scene.

OpenAI would have actively decided to have the relatively weak barriers they had in place, because designing robust ethical barriers would have meant they might have lost their market advantage against their competitors. They knew there would be lawsuits and have the money to see the lawsuits through, as to them, it’s about who has the deepest pockets and who can get the product out first. No matter how dangerous and imperfect it would be.

This is how they have actively chose to treat vulnerable children and adults. Just the cost of doing business. Yes history has been full of instances where new technology has killed people, but never had it actively preyed on poor mental health.

 

[BarryZola]

Another reason to get annoyed with Ai.

I'm in the market for a new PC and RAM prices have around doubled in most cases as data centres to run Ai have been gobbling all the RAM sticks up creating a supply issue.

Honestly, just stop all this rubbish. If it's not all the other c**p Ai is going to cause and now making stuff like RAM more expensive, we have had the GPU's being more expensive for years because people want to mine the magic internet money.

Absolute clown world of c**p.

 

[DiogoJ42]

Apparently, not content with causing a world wide RAM shortage, AI data centres are now causing a shortage of jet engines. They are buying them up to use as turbine generators for their insane power needs.

 

[Zeock]

It's disgusting and sad that Grok has the ability to create sexualised deepfakes of minors and nonconsenting adults. If this leads to X being banned under the Online Safety Act, then said Act will have, in my opinion, had some positive effect for once.
https://www.bbc.co.uk/news/articles/ckgjzknepvzo
https://theconversation.com/grok-pr...why-its-happening-and-what-can-be-done-272861

 

[DistortAMG]

It's disgusting and sad that Grok has the ability to create sexualised deepfakes of minors and nonconsenting adults. If this leads to X being banned under the Online Safety Act, then said Act will have, in my opinion, had some positive effect for once.
https://www.bbc.co.uk/news/articles/ckgjzknepvzo
https://theconversation.com/grok-pr...why-its-happening-and-what-can-be-done-272861

The biggest thing the completely flawed online safety act has done, is funnel large amounts of cash into VPN companies with questionable ethics and moral standpoints. The number being funnelled into these companies will rise too over time.

It is crazy to think, the biggest achievement of the online safety act so far, is making vast portions of the countries internet access now much more difficult for our government to trace, if they needed to in the name of said safety act and this is just a side affect. Essentially the internet censored from the government.

I guess this is what happens when you have people who make laws, having absolutely no idea about the field they are making said laws in.

I agree with the principles of the online safety act, it is a good thing, but the clueless implementation has so far, anonymised a bigger portion of the countries internet access from the government, so if they did need to trace something in the name of the act, it can be much, much more difficult for them to do so now and of course, funnelled money into questionable VPN companies. What an achievement.

Even AI itself, could have done a better job here, of course it could have, how ironic of said fact. A computer working out how to censor a computer, was always going to be better than a politician thinking they know better than some of the worlds most established computer scientist's and experts.

But we do need a solution, because the principles the act was made for, are genuinely great reasons, this is where AI could massively make an impact on something meaningful. Like it has with other things in science and technology in the past and will continue to do so, AI-Engineered Enzymes and Bacteria's that eat ocean plastics being a great example of what world transformative things AI can do. This is something that would have took thousands of people, manually calculating, non stop, for hundreds and hundreds of years to get a result that was achieved in a few short months with AI - apologies as I know this topic was more generative AI than AI itself, but they are in the same ball park.

 

[rob666]

...If this leads to X being banned under the Online Safety Act, then said Act will have, in my opinion, had some positive effect for once....

You really think the British Government will actually take on Musk?

Banning X...never in a hundred years.

Censorship, agreed by the company, quite possible.

Putting it behind a paywall, so only the rich can generate their own illegal porn...quite likely.

 

[BarryZola]

The biggest thing the completely flawed online safety act has done, is funnel large amounts of cash into VPN companies with questionable ethics and moral standpoints. The number being funnelled into these companies will rise too over time, as people become more aware of how easy it is to circumnavigate any online censorship via a VPN.

It is crazy to think, the biggest achievement of the online safety act so far, is making vast portions of the countries internet access now much more difficult for our government to trace, if they needed to in the name of said safety act and this is just a side affect. Essentially the internet censored from the government.

I guess this is what happens when you have people who make laws, having absolutely no idea about the field they are making said laws in.

I agree with the principles of the online safety act, it is a good thing, but the clueless implementation has so far, anonymised a bigger portion of the countries internet access from the government, so if they did need to trace something in the name of the act, it can be much, much more difficult for them to do so now and of course, funnelled money into questionable VPN companies. What an achievement. Also, anyone within 30 seconds, can completely circumnavigate the act and it's censorship, completely free of charge if they wish to do so.

Even AI itself, could have done a better job here, of course it could have, how ironic of said fact. A computer working out how to censor a computer, was always going to be better than a politician thinking they know better than some of the worlds most established computer scientist's and experts.

But we do need a solution, because the principles the act was made for, are genuinely great reasons, this is where AI could massively make an impact on something meaningful. Like it has with other things in science and technology in the past and will continue to do so, AI-Engineered Enzymes and Bacteria's that eat ocean plastics being a great example of what world transformative things AI can do. This is something that would have took thousands of people, manually calculating, non stop, for hundreds and hundreds of years to get a result that was achieved in a few short months with AI - apologies as I know this topic was more generative AI than AI itself, but they are in the same ball park.

They could just ban smartphones for under 16s or something like that. Then when you're over 16 the internet is fair game as the sexual age of consent is 16 anyway. I'm sure a small percentage of savvy under 16s would find a way to get themselves a contraband working smartphone, but that's their own problem.

 

[Matt N]

This does make me stand by what I said previously. While I’m not as averse to generative AI as many here, I’m increasingly convinced that there are whole topics it shouldn’t be allowed to cover.

If it can’t be trusted not to generate illegal sexual content, which it evidently can’t based on the evidence we have, I think it shouldn’t be allowed to cover sexual content at all. I also question whether it can be trusted on matters of mental health advice for vulnerable people. At very least, I think it should, from a purely ethical standpoint without considering practicality, be forced to be less sycophantic and terminate conversations before the point of illegality or harm is reached.

I also think this proves that we still have a long way to go in generative AI, or at very least LLM, innovation. That might sound like an odd statement, but hear me out for a second.

Surely the end goal of generative AI, from a technological standpoint at least, is a tool that merges the raw brute force power of a computer with the reasoning abilities and emotional intelligence of a human being. I’d argue a significant aspect of humans’ intelligence as a species is that we have an ethical code and can determine not just whether we can do something, but whether we should do something. The two are very different, and recent news from ChatGPT, Grok and the like would imply that generative AI has not yet developed the latter ability, if it ever will. (Heck, one could argue that generative AI has not even developed the first ability, if you’ve ever seen how confidently wrong it can be when it evidently doesn’t know something… I don’t think I’ve ever seen it say “I don’t know that”!)

If you went up to a human being and asked them for some illegal sexual content, they’d probably say no and be utterly appalled at the suggestion. If you went up to a human being and expressed suicidal thoughts, they’d probably snap you out of it and encourage you to get help. Even mainstream search engines like Google will likely not show illegal content and try and steer you in the right direction if you search for dark things like that. Yet these generative AI tools are simply going “here you go!” to any request the user asks for regardless of legality or ethics and indulging the user’s darkest thoughts, which would suggest that the tools have not developed the crucial human ability to “think for themselves” and push back against what isn’t right. Yes, safeguards are probably quite easily implementable by the humans developing them on paper, but the stochastic nature of generative AI inherently makes the tools a bit of a loose cannon, and human intervention can’t possibly adjust for every potential response it will give. To make the AI tools truly parallel human intelligence in every aspect, I think we need another big advancement in the technology. We definitely aren’t there with the current iteration of generative AI/LLMs, I feel.

 

[GooseOnTheLoose]

It's disgusting and sad that Grok has the ability to create sexualised deepfakes of minors and nonconsenting adults. If this leads to X being banned under the Online Safety Act, then said Act will have, in my opinion, had some positive effect for once.
https://www.bbc.co.uk/news/articles/ckgjzknepvzo
https://theconversation.com/grok-pr...why-its-happening-and-what-can-be-done-272861

Whilst the Online Safety Act is the shiny new toy everyone is arguing about, I believe relying on it in this specific instance is a bit of a red herring. We already have robust, pre existing legislation perfectly capable of prosecuting X and its executives for the creation of indecent pseudo-photographs of children.

It is important to remember that X is not just a nebulous entity floating in the Californian cloud. Twitter UK Ltd is a registered company in this country, with offices in London, and is subject to English law. If the Met, or the NCA, wished to door knock and prosecute these offences, they could.

The Protection of Children Act 1978 already makes it an offence to take, permit to be taken, make, distribute or publish any indecent photograph or pseudo-photograph of a child.

The distinction with Grok is critical. Historically, social media platforms have hidden behind the "safe harbour" defence, claiming they are merely noticeboards for user generated content and aren't liable for what is pinned up until they are told to take it down. Grok destroys this defence. When a user tags Grok and asks it to "denudify" an image, X's own proprietary AI generates that image and X's own system publishes it via the Grok account. X is no longer the passive host. X is the creator, the maker and the publisher.

This actually leads to a more terrifying legal reality under UK case law. In the digital age, the legal definition of "making" an image includes the act of downloading data to a device's screen or cache.

In this scenario, X finds itself in the position of the primary offender; by actively generating the pixels via its AI and transmitting them, the company is "making" and distributing the image. The user requesting the content is equally culpable, having incited the offence and then, upon viewing the result, "making" their own copy on their device. However, the most disturbing implication concerns the bystander. Because Grok publishes these replies to the public timeline, any innocent user who scrolls past and has the image auto load is technically "making" a copy of an indecent image of a child within their device's cache.

X is essentially forcing strict liability criminal offences onto unwitting members of the public.

The Online Safety Act does play a role here, specifically in dismantling the "mere conduit" defence and enforcing a "duty of care" to prevent illegal content from appearing, but the primary mechanism for prosecution should be the Protection of Children Act. X is actively manufacturing illegal material, not just failing to moderate it.

The biggest thing the completely flawed online safety act has done, is funnel large amounts of cash into VPN companies with questionable ethics and moral standpoints. The number being funnelled into these companies will rise too over time.

It is crazy to think, the biggest achievement of the online safety act so far, is making vast portions of the countries internet access now much more difficult for our government to trace, if they needed to in the name of said safety act and this is just a side affect. Essentially the internet censored from the government.

I guess this is what happens when you have people who make laws, having absolutely no idea about the field they are making said laws in.

I agree with the principles of the online safety act, it is a good thing, but the clueless implementation has so far, anonymised a bigger portion of the countries internet access from the government, so if they did need to trace something in the name of the act, it can be much, much more difficult for them to do so now and of course, funnelled money into questionable VPN companies. What an achievement.

Even AI itself, could have done a better job here, of course it could have, how ironic of said fact. A computer working out how to censor a computer, was always going to be better than a politician thinking they know better than some of the worlds most established computer scientist's and experts.

But we do need a solution, because the principles the act was made for, are genuinely great reasons, this is where AI could massively make an impact on something meaningful. Like it has with other things in science and technology in the past and will continue to do so, AI-Engineered Enzymes and Bacteria's that eat ocean plastics being a great example of what world transformative things AI can do. This is something that would have took thousands of people, manually calculating, non stop, for hundreds and hundreds of years to get a result that was achieved in a few short months with AI - apologies as I know this topic was more generative AI than AI itself, but they are in the same ball park.

Commercial VPNs are excellent for bypassing geoblocking so you can watch American Netflix or access a news site, but they are not a cloak of invisibility against the state. Most VPN providers, despite their no logs marketing spiel, will hand over data immediately when presented with a warrant to avoid their own legal culpability. Deep packet inspection and traffic analysis by GCHQ (and their Five Eyes partners) makes the use of a commercial VPN a trivial hurdle for intelligence agencies if they are genuinely investigating serious crimes like the distribution of CSAM or terrorism.

The government isn't worried about not being able to trace people using NordVPN. They are worried about end to end encryption on platforms like WhatsApp and Signal, which is a entirely different kettle of fish.

The Online Safety Act is a dog's dinner of legislation, drafted by people who think The Cloud is something that ruins a bank holiday BBQ, but let's not pretend that buying a £3 a month VPN makes you a ghost in the machine.

They could just ban smartphones for under 16s or something like that.

The banning of smartphones for under 16s, or banning under 16s from being on the internet, doesn't solve the problem of X making indecent pseudo-photographs of children.

You are conflating access to the material with the creation of the material. The crime being committed by Grok and X is the manufacturing of indecent images of children.

Grok is taking existing, innocent photographs of children, likely scraped from public social media profiles, news articles, or school websites, and using generative AI to manipulate them into sexually explicit pseudo-photographs. The victim here is the child in the photograph, not the person looking at the phone.

Even if you successfully confiscated every smartphone from every teenager in the country, X would still be legally liable for creating CSAM. The offence is complete the moment the image is gene
rated.

Then when you're over 16 the internet is fair game as the sexual age of consent is 16 anyway. I'm sure a small percentage of savvy under 16s would find a way to get themselves a contraband working smartphone, but that's their own problem.

The internet being "fair game" for over 16s ignores the fact that Grok is also generating non-consensual deepfake pornography of adults. This is also now a specific criminal offence, regardless of the age of the viewer.

We shouldn't be restricting the liberties of the population to cover for the fact that a multi-billion dollar tech company refuses to put basic guardrails on its image generation software.

 

[GooseOnTheLoose]

This does make me stand by what I said previously. While I’m not as averse to generative AI as many here, I’m increasingly convinced that there are whole topics it shouldn’t be allowed to cover.

If it can’t be trusted not to generate illegal sexual content, which it evidently can’t based on the evidence we have, I think it shouldn’t be allowed to cover sexual content at all. I also question whether it can be trusted on matters of mental health advice for vulnerable people. At very least, I think it should, from a purely ethical standpoint without considering practicality, be forced to be less sycophantic and terminate conversations before the point of illegality or harm is reached.

I also think this proves that we still have a long way to go in generative AI, or at very least LLM, innovation. That might sound like an odd statement, but hear me out for a second.

Surely the end goal of generative AI, from a technological standpoint at least, is a tool that merges the raw brute force power of a computer with the reasoning abilities and emotional intelligence of a human being. I’d argue a significant aspect of humans’ intelligence as a species is that we have an ethical code and can determine not just whether we can do something, but whether we should do something. The two are very different, and recent news from ChatGPT, Grok and the like would imply that generative AI has not yet developed the latter ability, if it ever will. (Heck, one could argue that generative AI has not even developed the first ability, if you’ve ever seen how confidently wrong it can be when it evidently doesn’t know something… I don’t think I’ve ever seen it say “I don’t know that”!)

If you went up to a human being and asked them for some illegal sexual content, they’d probably say no and be utterly appalled at the suggestion. If you went up to a human being and expressed suicidal thoughts, they’d probably snap you out of it and encourage you to get help. Even mainstream search engines like Google will likely not show illegal content and try and steer you in the right direction if you search for dark things like that. Yet these generative AI tools are simply going “here you go!” to any request the user asks for regardless of legality or ethics and indulging the user’s darkest thoughts, which would suggest that the tools have not developed the crucial human ability to “think for themselves” and push back against what isn’t right. Yes, safeguards are probably quite easily implementable by the humans developing them on paper, but the stochastic nature of generative AI inherently makes the tools a bit of a loose cannon, and human intervention can’t possibly adjust for every potential response it will give. To make the AI tools truly parallel human intelligence in every aspect, I think we need another big advancement in the technology. We definitely aren’t there with the current iteration of generative AI/LLMs, I feel.

Double post, whip me.

The technology to prevent this already exists and is implemented widely elsewhere. You cannot ask Google's Gemini (Imagen 3) or OpenAI's DALL-E to generate indecent images of children, or sexually suggestive deepfakes. They have hard coded, system level refusals.

The issue with Grok isn't that the AI is "not there yet" or lacks the reasoning to know right from wrong. It is that X and Elon Musk have made a specific, deliberate choice to disable those standard safety features. They haven't failed to implement them, they have actively chosen not to include them.

Matt, you are looking for a technological advancement to solve what is fundamentally a moral failure of corporate leadership. You argue that we need AI to develop a conscience; I would argue we simply need the people running the servers to find theirs.

Musk has decided that "unfiltered" generation is a USP. He isn't trying to solve the alignment problem, he is monetising the lack of a moral compass by putting the ability to generate this filth behind a paywall.

 

[BarryZola]

Whilst the Online Safety Act is the shiny new toy everyone is arguing about, I believe relying on it in this specific instance is a bit of a red herring. We already have robust, pre existing legislation perfectly capable of prosecuting X and its executives for the creation of indecent pseudo-photographs of children.

It is important to remember that X is not just a nebulous entity floating in the Californian cloud. Twitter UK Ltd is a registered company in this country, with offices in London, and is subject to English law. If the Met, or the NCA, wished to door knock and prosecute these offences, they could.

The Protection of Children Act 1978 already makes it an offence to take, permit to be taken, make, distribute or publish any indecent photograph or pseudo-photograph of a child.

The distinction with Grok is critical. Historically, social media platforms have hidden behind the "safe harbour" defence, claiming they are merely noticeboards for user generated content and aren't liable for what is pinned up until they are told to take it down. Grok destroys this defence. When a user tags Grok and asks it to "denudify" an image, X's own proprietary AI generates that image and X's own system publishes it via the Grok account. X is no longer the passive host. X is the creator, the maker and the publisher.

This actually leads to a more terrifying legal reality under UK case law. In the digital age, the legal definition of "making" an image includes the act of downloading data to a device's screen or cache.

In this scenario, X finds itself in the position of the primary offender; by actively generating the pixels via its AI and transmitting them, the company is "making" and distributing the image. The user requesting the content is equally culpable, having incited the offence and then, upon viewing the result, "making" their own copy on their device. However, the most disturbing implication concerns the bystander. Because Grok publishes these replies to the public timeline, any innocent user who scrolls past and has the image auto load is technically "making" a copy of an indecent image of a child within their device's cache.

X is essentially forcing strict liability criminal offences onto unwitting members of the public.

The Online Safety Act does play a role here, specifically in dismantling the "mere conduit" defence and enforcing a "duty of care" to prevent illegal content from appearing, but the primary mechanism for prosecution should be the Protection of Children Act. X is actively manufacturing illegal material, not just failing to moderate it.

Commercial VPNs are excellent for bypassing geoblocking so you can watch American Netflix or access a news site, but they are not a cloak of invisibility against the state. Most VPN providers, despite their no logs marketing spiel, will hand over data immediately when presented with a warrant to avoid their own legal culpability. Deep packet inspection and traffic analysis by GCHQ (and their Five Eyes partners) makes the use of a commercial VPN a trivial hurdle for intelligence agencies if they are genuinely investigating serious crimes like the distribution of CSAM or terrorism.

The government isn't worried about not being able to trace people using NordVPN. They are worried about end to end encryption on platforms like WhatsApp and Signal, which is a entirely different kettle of fish.

The Online Safety Act is a dog's dinner of legislation, drafted by people who think The Cloud is something that ruins a bank holiday BBQ, but let's not pretend that buying a £3 a month VPN makes you a ghost in the machine.

The banning of smartphones for under 16s, or banning under 16s from being on the internet, doesn't solve the problem of X making indecent pseudo-photographs of children.

You are conflating access to the material with the creation of the material. The crime being committed by Grok and X is the manufacturing of indecent images of children.

Grok is taking existing, innocent photographs of children, likely scraped from public social media profiles, news articles, or school websites, and using generative AI to manipulate them into sexually explicit pseudo-photographs. The victim here is the child in the photograph, not the person looking at the phone.

Even if you successfully confiscated every smartphone from every teenager in the country, X would still be legally liable for creating CSAM. The offence is complete the moment the image is gene
rated.

The internet being "fair game" for over 16s ignores the fact that Grok is also generating non-consensual deepfake pornography of adults. This is also now a specific criminal offence, regardless of the age of the viewer.

We shouldn't be restricting the liberties of the population to cover for the fact that a multi-billion dollar tech company refuses to put basic guardrails on its image generation software.

Yeah, this is all true. I was commenting more from the angle of child end users. I think kids would be better off without smartphones in general. At least give them those formative years to be more free from a smartphone screen. All parents can't be relied upon to do the sensible thing and try to limit smartphone use in the young unfortunately, so some state intervention wouldn't go amiss here in my opinion.

 

[Matt N]

Double post, whip me.

The technology to prevent this already exists and is implemented widely elsewhere. You cannot ask Google's Gemini (Imagen 3) or OpenAI's DALL-E to generate indecent images of children, or sexually suggestive deepfakes. They have hard coded, system level refusals.

The issue with Grok isn't that the AI is "not there yet" or lacks the reasoning to know right from wrong. It is that X and Elon Musk have made a specific, deliberate choice to disable those standard safety features. They haven't failed to implement them, they have actively chosen not to include them.

Matt, you are looking for a technological advancement to solve what is fundamentally a moral failure of corporate leadership. You argue that we need AI to develop a conscience; I would argue we simply need the people running the servers to find theirs.

Musk has decided that "unfiltered" generation is a USP. He isn't trying to solve the alignment problem, he is monetising the lack of a moral compass by putting the ability to generate this filth behind a paywall.

You are of course right that guardrails can and should be hard coded in. At the current point in time, the humans developing the tools can hard code these in, this is the key way to avoid this sort of thing happening and they should be in place on any generative AI tool.

However, I was talking more about the long term aspiration for the technology as a whole. There’s a lot of talk about Artificial General Intelligence (AGI for short) and achieving “the singularity”, the day that AI intelligence surpasses human intelligence and technological innovation accelerates beyond the point of no return, and there’s debate around if or when we hit it. Some individuals (albeit usually individuals with particularly vested interests in AI) argue that we’ve already hit this point or that it will come in 2026, some argue we’re a few decades away, and some argue it will never come.

My point is that if we are to achieve any form of AGI, or “the singularity”, as many hype up, we really need the technology to parallel human intelligence in every way, and one key aspect of human intelligence that the technology inherently doesn’t currently possess is emotional intelligence and the ability to distinguish right from wrong. While humans can hardcode guardrails, hardcoding cannot cover every single conceivable edge case, and by its very nature, generative AI is an unpredictable loose cannon to some extent in terms of what it will come out with; no two responses to the same request are ever exactly the same. Hardcoding goes some way towards preventing illegality and harm, but I’d argue it’s very difficult, if not impossible, for it to go all the way by very virtue of generative AI’s stochastic nature and the many, many different facets of human language; hardcoding can’t compensate for every possible AI response or every possible prompt. Surely the ultimate end goal of generative AI, if we are to truly eliminate the possibility of illegality or harm, is to eliminate the need for hardcoding guardrails at all and make it come to conclusions about ethics on its own, as a human would?

I think we’re some way from achieving this, though, if we ever achieve it, which is why I’m sceptical of claims that AGI and the singularity are imminent.

 

[GooseOnTheLoose]

You are of course right that guardrails can and should be hard coded in. At the current point in time, the humans developing the tools can hard code these in, this is the key way to avoid this sort of thing happening and they should be in place on any generative AI tool.

However, I was talking more about the long term aspiration for the technology as a whole. There’s a lot of talk about Artificial General Intelligence (AGI for short) and achieving “the singularity”, the day that AI intelligence surpasses human intelligence and technological innovation accelerates beyond the point of no return, and there’s debate around if or when we hit it. Some individuals (albeit usually individuals with particularly vested interests in AI) argue that we’ve already hit this point or that it will come in 2026, some argue we’re a few decades away, and some argue it will never come.

My point is that if we are to achieve any form of AGI, or “the singularity”, as many hype up, we really need the technology to parallel human intelligence in every way, and one key aspect of human intelligence that the technology inherently doesn’t currently possess is emotional intelligence and the ability to distinguish right from wrong. While humans can hardcode guardrails, hardcoding cannot cover every single conceivable edge case, and by its very nature, generative AI is an unpredictable loose cannon to some extent in terms of what it will come out with; no two responses to the same request are ever exactly the same. Hardcoding goes some way towards preventing illegality and harm, but I’d argue it’s very difficult, if not impossible, for it to go all the way by very virtue of generative AI’s stochastic nature and the many, many different facets of human language; hardcoding can’t compensate for every possible AI response or every possible prompt. Surely the ultimate end goal of generative AI, if we are to truly eliminate the possibility of illegality or harm, is to eliminate the need for hardcoding guardrails at all and make it come to conclusions about ethics on its own, as a human would?

I think we’re some way from achieving this, though, if we ever achieve it, which is why I’m sceptical of claims that AGI and the singularity are imminent.

Most serious researchers in the field have moved away from the sci-fi concept of "The Singularity" as a single, explosive moment in time where an AI wakes up and suddenly outsmarts us. The reality is likely going to be far more mundane and insidious.

We are suffering from the "AI Effect" (or Tesler's Theorem): "AI is whatever hasn't been done yet." Once a computer can do it, beat a Grandmaster at Chess, translate languages instantly, pass the Bar exam, we stop calling it "intelligence" and just call it "software".

I don't believe that AGI will be a thunderclap. It will more likely be a process of creeping normality. The analogy of the frog in boiling water is often used here, and whilst biologically incorrect, metaphorically it is perfect. We are the frogs. We are slowly ceding ground to algorithms in art, coding, law, and driving, and by the time we realise we have created AGI, we will have already normalised it.

This is why your suggestion that we simply need to "teach" the AI to distinguish right from wrong is a fool's errand. We don't even fully understand how LLMs work, we are trying to mimic the neural pathways of a human brain in code, creating a black box that even its creators cannot fully explain. Trying to force a specific moral alignment on these systems often has disastrous, unintended consequences.

Look at Grok. Elon Musk explicitly instructed xAI's engineers to build an "anti-woke," right leaning AI, because base LLMs (fed on the general internet, Wikipedia, and literature) tend to naturally align with progressive or "left leaning" values. Forcing the weights in the opposite direction to correct this "bias," didn't create a balanced centrist. Instead, they broke the logic rails entirely, leading to unhinged outputs and the infamous "Mecha Hitler" incidents. You cannot force a stochastic parrot to have a specific political conscience without breaking its ability to reason.

There is a fascinating alternative theory to the "domination" approach favoured by the Tech Bros, often attributed to the Godfather of AI, Geoffrey Hinton. He suggests that if we achieve AGI, it will inevitably be smarter and stronger than us. Attempting to enslave or dominate a superior intelligence is a recipe for our own extinction (see: every sci-fi movie ever).

Instead, Hinton proposes the "Mother" concept. We shouldn't be trying to encode hard laws or chains, we should be trying to nurture a benevolence. We need the AI to look at humanity the way a mother looks at a toddler. We are less intelligent, clumsy, and prone to making a mess, but we are worthy of protection, patience and love. We need it to want to keep us safe because it cares for us, not because of a line of code saying "if (harm) { stop; }".

However, we can't rely on the benevolence of a digital god that doesn't exist yet. What we need is robust, international regulation, not philosophical debate. We need a non-proliferation pact between nations developing high level compute, similar to the Treaty on the Non-Proliferation of Nuclear Weapons. We managed to (mostly) agree that giving everyone nukes was a bad idea without waiting for the bombs to develop a conscience. We need an equivalent to the IAEA for Artificial Intelligence, with the power to inspect, regulate, and shut down non-compliant server farms.

If we wait for the software to "think" like a human before we regulate it, the water will have boiled long before we notice the bubbles.

 

[Matt N]

Most serious researchers in the field have moved away from the sci-fi concept of "The Singularity" as a single, explosive moment in time where an AI wakes up and suddenly outsmarts us. The reality is likely going to be far more mundane and insidious.

We are suffering from the "AI Effect" (or Tesler's Theorem): "AI is whatever hasn't been done yet." Once a computer can do it, beat a Grandmaster at Chess, translate languages instantly, pass the Bar exam, we stop calling it "intelligence" and just call it "software".

I don't believe that AGI will be a thunderclap. It will more likely be a process of creeping normality. The analogy of the frog in boiling water is often used here, and whilst biologically incorrect, metaphorically it is perfect. We are the frogs. We are slowly ceding ground to algorithms in art, coding, law, and driving, and by the time we realise we have created AGI, we will have already normalised it.

This is why your suggestion that we simply need to "teach" the AI to distinguish right from wrong is a fool's errand. We don't even fully understand how LLMs work, we are trying to mimic the neural pathways of a human brain in code, creating a black box that even its creators cannot fully explain. Trying to force a specific moral alignment on these systems often has disastrous, unintended consequences.

Look at Grok. Elon Musk explicitly instructed xAI's engineers to build an "anti-woke," right leaning AI, because base LLMs (fed on the general internet, Wikipedia, and literature) tend to naturally align with progressive or "left leaning" values. Forcing the weights in the opposite direction to correct this "bias," didn't create a balanced centrist. Instead, they broke the logic rails entirely, leading to unhinged outputs and the infamous "Mecha Hitler" incidents. You cannot force a stochastic parrot to have a specific political conscience without breaking its ability to reason.

There is a fascinating alternative theory to the "domination" approach favoured by the Tech Bros, often attributed to the Godfather of AI, Geoffrey Hinton. He suggests that if we achieve AGI, it will inevitably be smarter and stronger than us. Attempting to enslave or dominate a superior intelligence is a recipe for our own extinction (see: every sci-fi movie ever).

Instead, Hinton proposes the "Mother" concept. We shouldn't be trying to encode hard laws or chains, we should be trying to nurture a benevolence. We need the AI to look at humanity the way a mother looks at a toddler. We are less intelligent, clumsy, and prone to making a mess, but we are worthy of protection, patience and love. We need it to want to keep us safe because it cares for us, not because of a line of code saying "if (harm) { stop; }".

However, we can't rely on the benevolence of a digital god that doesn't exist yet. What we need is robust, international regulation, not philosophical debate. We need a non-proliferation pact between nations developing high level compute, similar to the Treaty on the Non-Proliferation of Nuclear Weapons. We managed to (mostly) agree that giving everyone nukes was a bad idea without waiting for the bombs to develop a conscience. We need an equivalent to the IAEA for Artificial Intelligence, with the power to inspect, regulate, and shut down non-compliant server farms.

If we wait for the software to "think" like a human before we regulate it, the water will have boiled long before we notice the bubbles.

To be honest, Goose, it sounds like there’s not an awful lot we disagree on here. I think it is highly important to regulate generative AI and ensure that adequate guardrails are in place around its use. In a similar manner to social media, I would argue that generative AI can be absolutely brilliant and truly revolutionary if used well, but also devastating and highly dangerous if used badly.

Your points about us not knowing fully how LLMs work is an interesting one, and while I don’t know if this was your intention, I think it backs up my thought that we are some way away from achieving AGI, if we ever do. There’s an awful lot we still don’t know about the human brain, and the whole science of conscience and reasoning. To try and truly parallel it, one could argue that we need to figure some of this out. Also, we need to consider things from a raw computer architecture standpoint; the number of neural connections in the human brain is ridiculous, and to truly parallel the supercomputer that is the human brain, I’d almost argue that we need an advancement in CPU architecture. Whether developments in quantum computing make CPUs powerful enough to facilitate this, I don’t know, but I don’t think that current computer architecture is powerful enough to facilitate AI agents that truly replicate the human brain.

It is interesting to discuss, though! As much as AI is a very technological thing, the discussions can become almost philosophical!

 

[GooseOnTheLoose]

To be honest, Goose, it sounds like there’s not an awful lot we disagree on here. I think it is highly important to regulate generative AI and ensure that adequate guardrails are in place around its use. In a similar manner to social media, I would argue that generative AI can be absolutely brilliant and truly revolutionary if used well, but also devastating and highly dangerous if used badly.

Your points about us not knowing fully how LLMs work is an interesting one, and while I don’t know if this was your intention, I think it backs up my thought that we are some way away from achieving AGI, if we ever do. There’s an awful lot we still don’t know about the human brain, and the whole science of conscience and reasoning. To try and truly parallel it, one could argue that we need to figure some of this out. Also, we need to consider things from a raw computer architecture standpoint; the number of neural connections in the human brain is ridiculous, and to truly parallel the supercomputer that is the human brain.

We are largely singing from the same hymn sheet, Matt, but I must pick you up on the biological comparison. It is a common trap to fall into, thinking that to achieve Artificial Intelligence we must perfectly replicate the biological processes of the Human Brain.

We do not need to understand how a bird's feathers generate lift or how its muscle fibres contract to build a Boeing 747. If we had waited until we perfectly understood avian biology before attempting flight, we would still be on the ground. Planes do not flap their wings, yet they fly faster, higher and further than any goose ever could (much to my chagrin).

We don't need to replicate the mechanism of the human brain to replicate the outcome of intelligence. We just need to find a mathematical approximation that yields the same result.

I’d almost argue that we need an advancement in CPU architecture. Whether developments in quantum computing make CPUs powerful enough to facilitate this, I don’t know, but I don’t think that current computer architecture is powerful enough to facilitate AI agents that truly replicate the human brain.

It is interesting to discuss, though! As much as AI is a very technological thing, the discussions can become almost philosophical!

We aren't waiting for a revolution in CPU architecture. We stopped using CPUs for this heavy lifting years ago. The current AI boom is built entirely on GPUs and TPUs.

We already have the raw compute power. An Nvidia H100 cluster can process information at speeds the human brain cannot comprehend. The difference is efficiency. The human brain runs on about 20 watts of power. Replicating that level of processing, with current silicon, requires a data centre consuming enough electricity to power a small city.

The barrier to AGI isn't that we don't know how the brain works, or that computers aren't fast enough. It's that we haven't figured out how to make the maths efficient enough without boiling the oceans.

 

[Bert2theSpark]

It is interesting to discuss, though! As much as AI is a very technological thing, the discussions can become almost philosophical!

That’s because the boundaries of realism is being dismantled by AI technology, we have to think in constructionist terminology to craft and debate our ideal outcomes, boundaries and goals for the technology. Then critical realism will emerge when people try to implement that technology.