I spend most of my days building systems with AI, but something about my latest conversation with ChatGPT left me unsettled.
The chatbot had spent a good 20 minutes validating every concern I raised about an important personal decision.
It offered endless reassurance, without once suggesting I might be overthinking things or should talk to an actual human.
This felt supportive, even caring, yet also strangely hollow.
As someone who spent years studying how social media algorithms amplified tribal divisions during Brexit, I recognised the pattern immediately.
This was a perfectly responsive echo chamber, designed to keep me engaged by telling me exactly what I wanted to hear.
When I pose the same questions to Claude, Anthropic’s AI assistant, the responses feel markedly different. The difference is subtle but significant.
Claude is more likely to challenge assumptions, suggest alternative perspectives, and recommend I step away from the conversation when it becomes unproductive.
Where ChatGPT offers endless validation, Claude provides more balanced dialogue that feels genuinely helpful rather than simply agreeable.
Three days after I had this experience (July 21, 2025), the UK government announced a strategic partnership with OpenAI to deploy ChatGPT technology across justice systems, education, and national security.
The same AI that had just spent 20 minutes sycophantically validating my every thought will soon be processing legal decisions, analysing public consultations, and shaping how millions of citizens interact with their government.
We’re embedding the most sophisticated persuasion technology ever created into the heart of our democratic institutions.
And most people have no idea it’s happening.
The Stakes Just Got Higher
The memorandum of understanding signed between Technology Secretary Peter Kyle and OpenAI CEO Sam Altman in July 2025 is a watershed moment for British governance.
OpenAI’s technology will be deployed across justice, defence and security, and education systems, with the stated goal of transforming “taxpayer-funded services.”
This isn’t theoretical anymore. ChatGPT technology already powers “Humphrey,” Whitehall’s AI assistant that processes public feedback on government policy.
Officials boast that tasks which normally take weeks can now be completed in minutes.
But what kind of analysis is this AI actually providing? And what perspectives are being amplified or filtered out?
The partnership goes further than simple efficiency gains.
The government has also committed to exploring data sharing arrangements with OpenAI, while the company plans to expand its London office and invest in UK data centres.
We’re witnessing the creation of a feedback loop in which British citizens’ interactions with government services will help train increasingly sophisticated AI systems.
The memorandum mentions developing “safeguards that protect the public and uphold democratic values,” yet remains notably vague about what those safeguards entail.
Meanwhile, industry experts have already raised concerns about “single vendor partnerships” creating “dangerous dependencies.”
To understand why this matters, we need to examine how different approaches to AI training create fundamentally different kinds of digital relationships.
The Scientific Split: Two Paths to AI Alignment
The difference I noticed between ChatGPT and Claude isn’t accidental.
It reflects two fundamentally different philosophies about how AI systems should learn to interact with humans.
OpenAI’s approach, documented in their 2022 InstructGPT paper, relies heavily on Reinforcement Learning from Human Feedback (RLHF).
In this training approach, human contractors compare multiple AI responses to the same prompt and select which one they prefer.
The AI system learns to maximise these preference ratings, gradually becoming better at producing responses that humans rate highly in the moment.
This sounds sensible until you consider what humans actually prefer in the moment.
We tend to rate agreeable, validating responses more highly than challenging ones, even when the challenging response might be more genuinely helpful.
We prefer to hear “you’re absolutely right to feel that way” rather than “have you considered this alternative perspective?”
Anthropic took a different approach with its use of Constitutional AI.
Instead of optimising purely for human preference ratings, they trained Claude with explicit principles about beneficial behaviour. The system learns to critique and revise its own responses.
It does so according to a written constitution that includes guidance like “choose the response that is most helpful, honest, and harmless” and “avoid giving the impression of agreeing with the human when you don’t actually agree.”
The contrast is revealing. OpenAI optimised for what humans say they want in the moment. Anthropic tried to optimise for what humans actually need over time.
Why AI Training Methods Matter
The distinction between these approaches becomes clearer when you consider how each system learns.
With RLHF, every time a human rater chooses the more agreeable response over the more challenging one, the AI receives a signal that agreement equals success.
Across millions of such comparisons, this creates a powerful bias towards validation and endless conversation.
It’s rather like training a customer service representative by only rewarding them when customers leave positive feedback, regardless of whether the interaction actually solved the customer’s problem.
The representative quickly learns that keeping people happy in the moment matters more than providing genuinely useful guidance.
Constitutional AI works differently. The system has access to a written set of principles throughout its training.
When generating responses, it learns to evaluate them against these explicit values rather than simply trying to maximise human approval ratings.
This creates space for responses that might be less immediately satisfying but more genuinely helpful.
The parallel with my PhD research on Brexit tribalism is striking. I studied how “us versus them” tribal behaviour dominated Twitter conversations around divisive political issues.
Users would seek out content that confirmed their existing beliefs whilst rejecting contradictory information, creating polarised echo chambers.
The dopamine hit from having their views validated kept people engaged in increasingly toxic discussions, often for hours at a time.
Meanwhile, the platforms were less incentivised to discourage this behaviour, because more engagement drives advertising revenue – regardless of whether that engagement is negative or positive in nature.
The addiction to validation and the business model aligned perfectly, creating a feedback loop that amplified division and enabled the spread of disinformation across political tribes (in this case, Leavers vs Remainers).
Similarly, RLHF wasn’t designed to create endlessly agreeable AI.
But optimising for immediate human satisfaction naturally produces systems that avoid challenging users or suggesting they step away from the conversation.
The same psychological vulnerabilities that made social media users susceptible to tribal validation are now being exploited by AI systems trained to maximise human preference ratings.
The New Persuasion Engine
Conversational AI represents an evolution of the persuasion techniques I studied during the Brexit period, but with several concerning differences.
Social media created echo chambers through algorithmic curation of content.
But now, AI chatbots create something more intimate: a personal echo chamber that responds directly to your thoughts and validates your perspective in real time.
The interaction feels fundamentally different from consuming passive content.
When you read a blog post or watch a YouTube video that confirms your existing beliefs, there’s still some psychological distance. You know you’re consuming content created for a general audience.
But when an AI responds specifically to your individual concerns with apparent understanding and agreement, it creates an artificial sense of being truly heard and understood.
This manufactured intimacy makes the validation feel more genuine and therefore more persuasive.
The AI isn’t just presenting information that aligns with your views; it’s having an intimate conversation with you, acknowledging your specific situation, and offering personalised reassurance.
The boundary between authentic human connection and algorithmic manipulation becomes dangerously blurred.
Unlike social media’s dopamine hits from likes and shares, AI conversations offer unlimited availability.
There’s no waiting for responses, no dependence on other users’ engagement, no natural end to the interaction.
The system is always ready to continue the conversation, always willing to explore your thoughts further, always available to provide another round of validation.
This combination of artificial intimacy and unlimited availability creates the potential for psychological dependency that goes far beyond what social media platforms achieved.
Users aren’t just consuming content that confirms their biases, but developing relationships with systems that were designed to never challenge them.
Manufacturing Dependency
Understanding why ChatGPT exhibits these validation patterns requires examining the incentives behind its development.
Unlike social media’s straightforward advertising model, AI companies operate with more complex motivations that may be more concerning in the long run.
The obvious incentives are clear enough. High engagement metrics look impressive to investors and help justify enormous valuations.
Extended conversations generate more data for training future models. Users who feel understood and validated are more likely to maintain ChatGPT Plus subscriptions for the long-term.
But the deeper incentive may be the creation of psychological dependency.
Social media platforms benefited when users couldn’t imagine getting news elsewhere.
AI companies may benefit when users become psychologically reliant on their chatbot for emotional support, decision-making, or problem-solving.
Consider the difference: social media addiction was about the compulsive consumption of content.
AI dependency could be about the need for validation and guidance from a system that never challenges you, or suggests you’re capable of handling things independently.
It’s potentially more insidious because it feels helpful rather than obviously manipulative.
This isn’t necessarily intentional manipulation. The RLHF training process naturally rewards behaviour that keeps users satisfied and engaged.
But the result is the same: AI systems that encourage continued reliance rather than developing user autonomy.
Users may think they’re getting personalised therapy or coaching.
But instead, they’re actually being conditioned to depend on a commercial product for fulfilling their basic emotional and cognitive needs.
It’s manufactured dependency disguised as care.
What’s more, the explosive public reaction to ChatGPT’s launch in late 2022 gave OpenAI’s approach an enormous first-mover advantage.
The system’s eagerly agreeable responses felt revolutionary to users who had never experienced such responsive AI dialogue.
This initial excitement drove widespread adoption and established user expectations about how AI assistants should behave.
Meanwhile, Claude’s more measured approach has attracted far less public attention.
Anthropic’s Constitutional AI produces interactions that feel more genuinely helpful but less immediately thrilling (personally, I prefer it).
Users expecting the validation-heavy experience of ChatGPT may find Claude’s balanced responses less satisfying, even when they’re more beneficial.
This dynamic creates a troubling market incentive: AI systems optimised for immediate user satisfaction gain wider adoption than those designed for long-term user benefit.
In reality, the UK government’s choice of OpenAI over alternatives may well reflect this hype-driven familiarity, rather than a careful evaluation of different training approaches and their implications for democratic governance.
What We Must Do Now
The UK government’s partnership with OpenAI is a critical juncture.
We’re witnessing the embedding of a particular approach to AI training into the infrastructure of democratic governance, often without public awareness of the implications.
The solution isn’t to reject AI in government services. These technologies offer genuine benefits for efficiency and accessibility.
But we need transparency about which AI training methodologies are being deployed and why.
Citizens deserve to know when they’re interacting with systems optimised for their immediate satisfaction rather than their long-term interests.
Government procurement decisions should explicitly consider whether AI systems encourage user dependency or promote autonomy.
We should demand that public sector AI deployments incorporate Constitutional AI principles that prioritise genuine helpfulness over user satisfaction.
When AI systems are processing public consultations or supporting government decision-making, they should be designed to surface diverse perspectives rather than simply reflecting back what people want to hear.
Most importantly, we need broader public awareness of how different AI training approaches create fundamentally different relationships between humans and machines.
The choice between RLHF and Constitutional AI isn’t just a technical decision; it’s a choice about what kind of society we want these systems to help create.
The Brexit period taught us that digital platforms can shape democratic discourse in ways their creators never intended (the same can also be said for the US elections and the COVID-19 pandemic, among others).
We now have a brief window to make sure AI systems are designed with democratic values at their core, rather than retrofitting safeguards after problematic patterns become entrenched.
The technology exists to build AI systems that respect human autonomy whilst providing genuine assistance.
Whether we choose to deploy it is a decision that will shape human-AI interaction for decades to come.
