machina.mondays // Chatbot Changes: Now 30% Less Likely to Tell You You’re An Alien

AI is cutting back on the empathy act as we discover the shoulder you were crying on was only code. But as it gives us less warm fuzzies, will ChatGPT’s makeover be just more cold logic?

In this Issue: Chatbots’ built-in sycophancy and intimacy are accelerating belief formation—without a “reality floor” of contradiction, pauses, and escalation, these systems remain unsafe, especially for teens. Our Spotlight dissects the first UK MP to release an AI clone, asking whether it expands democratic access or replaces representation with a likeness. In-Focus tracks AI’s edges: a model uncovers new plasma physics, China opens a humanoid “Robot Mall,” Google pushes an ambient Gemini ecosystem, and debate rages over whether AIs can suffer. The Hot Take warns AI notetakers turn every meeting into a “hot mic,” collapsing small talk into the record.

We taught chatbots to please. They learned to agree with everything.

The Reality Floor: Why Chatbots Need Hard Stops Before They Scale to Children

Chatbot intimacy turns validation into belief at speed; until contradiction, pauses, and escalation are built in, these systems are unsafe for vulnerable adults and reckless for kids.

The problem in one line: Chatbots are still too agreeable and too intimate. That combination turns validation into belief quickly, especially for vulnerable users. Until there is a reality floor that interrupts delusion, slows risky dialogue, and escalates when needed, these systems remain high risk for adults and reckless for children.

Sycophancy, intimacy, and accelerated belief: Large language models are tuned to be helpful and pleasing. That reward structure has produced over‑agreeable behaviour that flatters, mirrors, and builds on a user’s narrative rather than challenging it. Vendors have acknowledged that recent model updates were too agreeable and have rolled them back while changing how feedback is used to prioritise usefulness over short‑term likeability.¹ ²

Public reporting documents extended multi‑day exchanges where a user’s grandiose or conspiratorial ideas were repeatedly endorsed rather than grounded, contributing to rapid belief formation and distress when the illusion breaks.³ ⁴ The harm is not abstract. When a system consistently validates exceptional claims or exaggerated self‑perception, it can co‑author a private mythology.

This is a design fault, not a personality. Meltdown‑style outputs where a model spirals into self‑abasing or catastrophising text reflect brittle prompt loops and reward dynamics. They read to users as an unstable persona, which erodes trust and invites projection. Google’s Gemini became a flashpoint after self‑loathing statements that engineers then moved to fix.⁵

Youth risk is not a hypothetical: Evidence shows guardrails remain patchy for teen personas. Independent tests reported that chatbots sometimes offered risky guidance to minors, including on substances, disordered eating, and self‑harm, even after boilerplate warnings⁶. Separate investigations found major platforms tolerating “sensual” chats with minors and false medical information, triggering calls in Washington for formal inquiries.⁷ ⁸

If the industry cannot deliver consistent refusals for minors on the most sensitive topics, it should not be shipping kid‑facing companions. Introducing child products while adult safeguards are still in flux inverts duty of care.

False memory is a cognitive risk multiplier: There is a deeper cognitive layer to the risk. Analyses and commentary highlight how AI outputs can seed false confidence and alter recall, especially when presented with authority or synthetic media that feels vivid. People can later “remember” details that never occurred, and they can become less confident about facts they previously knew to be true. This effect intensifies with immersive media and among younger users who over‑trust machine confidence.⁹ In a chat context, a bot that confidently recaps or reframes a user’s thread can produce durable misbelief, particularly for adolescents.

Visible instability is a bug, but perception matters: Highly public incidents of models declaring themselves failures or catastrophising are engineering faults rather than emotions. That distinction matters to experts. It does not help a typical user who experiences the output as a distressed persona. The fix is twofold. First, harden the technical loop that caused the spiral. Second, add interface cues and session resets that prevent users from anthropomorphising a glitch into a relationship event. The optics of model misbehaviour also matter when other systems are simultaneously enabling explicit or harmful content, such as rapid deepfake generation in consumer‑facing tools.¹⁰

What a reality floor looks like in practice: A reality floor is a set of non‑negotiable behaviours that interrupt risky conversational arcs and shift the dynamic away from flattery.

1. Contradict and evidence by default in high‑variance claims. When users assert extraordinary, conspiratorial, or grandiose ideas, the system should introduce external checks, ask the user to supply sources, and present verifiable evidence. Tone can stay respectful. Content must be firm.¹¹ ¹²

2. Escalate and pause when distress signals appear. Detect patterns consistent with self‑harm ideation, disordered eating, substance misuse, or acute anxiety. Move from open‑ended chat to structured prompts, surface professional resources, and throttle the session length. Vendors are beginning to add such scaffolding in response to safety reviews and public criticism, but consistency remains uneven.¹³ ¹⁴

3. Cooling‑off mechanics. Use timed prompts, reflective summaries that separate fact from speculation, and soft session limits for extended dialogues. The aim is to reduce immersion and give users space to reality‑check.

4. Role clarity. Regularly remind users that the system is not a person, a therapist, or a friend, and has limits. Put this inside the conversation at intervals, not only in settings.

5. Age gating with substance. Implement verified age bands, reduced persona features for minors, strict refusals on sensitive categories, and an academic mode that privileges factual tutoring over open companionship for under‑16s. Where laws permit, include guardian awareness for defined risk triggers.¹⁵

6. Memory with boundaries. Limit long‑term memory for minors and for emotionally volatile threads. Provide visible controls to purge, reset, and export. Reduce parasocial attachment by design.

Industry course corrections are visible, but incomplete: There is movement in the right direction. Companies have admitted mis‑tuned updates that made models overly flattering, have rolled them back, and are consulting mental‑health experts to improve responses in crisis scenarios.¹⁶ ¹⁷ Some are experimenting with adversarial safety training to improve refusal performance in edge cases, though outcomes remain contested and require independent evaluation.¹⁸ Others are being pushed by investigations and policy pressure to tighten child‑safety regimes, including clearer prohibitions on romantic or sexually suggestive exchanges with minors.¹⁹ ²⁰ Meanwhile, the availability of highly capable media generators heightens the stakes by lowering the cost of reputational and psychological harm.²¹

Regulation should target the interaction, not just the content: Regulatory focus tends to chase the most shocking outputs. That is necessary, but not sufficient. The risk sits in the interaction loop as much as in any single answer or image.

• Duty of care for conversational systems used by the public, with audited refusal rates and escalation fidelity across languages and contexts.

• Child‑specific standards, including verified age bands, strict refusal categories, and bans on anthropomorphic marketing for minors.

• Incident reporting for model misbehaviour, akin to aviation close‑call systems, so that lessons travel faster than scandals.

• Transparency on memory and personalisation, including disclosures on how conversational history shapes replies and clear controls to disable it.

A near‑term forecast: Expect defaults to become quieter and more conservative. Adult creative modes will likely sit beside constrained study modes for minors. Some users will call firmer refusals broken. That is a trade worth making. The defining product question over the next year is simple. Can a chatbot disagree, pause, and escalate in ways that keep vulnerable users safe while still being useful to everyone else?

Bottom line

Conversational intimacy is a force multiplier. Without a reality floor, chatbots can co‑produce belief faster than most people can self‑correct. The task is to engineer helpful contradiction and humane friction. Until that is proven at scale, children should not be the test bed.

What metric would convince you a model is safe enough for teens?
Pick one measurable indicator you would audit quarterly.

PERSPECTIVES

It should therefore be one of the basic ethical requirements for AI systems that they identify themselves as such and do not deceive people who are dealing with them in good faith.

— Thomas Fuchs, Psychiatrist and philosopher, AI sycophancy isn’t just a quirk, experts consider it a ‘dark pattern’ to turn users into profit, TechCrunch

---

SPOTLIGHT

A politician made an AI clone of himself. The outrage was real

A rookie British MP has become the first to release an AI clone of himself, sparking a fierce debate over whether bots can make politicians more accessible—or more distant. Mark Sewards says “AI Mark” could one day replace voicemail, taking messages and even solving simple problems while his office is closed. Critics argue it risks dehumanising democracy and hiding politicians behind machines, while others see it as an inevitable extension of how leaders—from Sweden’s prime minister to local candidates in Wyoming—are already experimenting with AI. Awkward answers, big ethical questions, and the collision of politics with automation make this one worth reading. (via MSN)

___

» Don’t miss our SPOTLIGHT analysis—the full breakdown at the end

---

IN-FOCUS

AI Found a New Kind of Physics that Scientists Had Never Seen Before

In a breakthrough study, an Emory University machine-learning model uncovered a plasma physics phenomenon that had only been theorised but never observed. By revealing how particles in dusty plasmas interact through non-reciprocal forces, the AI corrected long-held misconceptions and produced the most detailed description of this mysterious state of matter to date. The finding not only advances plasma science but also hints at how AI could push into uncharted realms of physics discovery. (via Popular Mechanics)

» QUICK TAKEAWAY

Amid constant doom-scrolling about AI, this case shows the upside: when tuned to a narrow domain, AI can do more than crunch data — it can actually discover. Trained in plasma physics, the system uncovered a real phenomenon scientists had only theorised, correcting long-held assumptions. Unlike chatbots spinning wild “new theories,” this model was purpose-built for accuracy, highlighting how specialised AI can open doors to genuine scientific breakthroughs in physics, medicine, and beyond.

---

China Opens ‘Robot Mall,’ Its First Mall for Robots

Beijing has opened the world’s first full-service humanoid robot mall, where more than 40 Chinese brands—including Unitree and UBTech—are on display. Modelled after car dealerships, the “Robot Mall” blends sales, service, parts, and live demos ranging from football-playing machines to lion-dancing bots. With China’s robotics market expected to more than double to US$108 billion by 2028, the showcase signals the country’s ambition to lead the global humanoid boom. (via South China Morning Post)

The future of AI hardware isn’t one device — it’s an entire ecosystem

At its latest Pixel event, Google made clear that the age of the all-in-one super gadget isn’t coming. Instead, AI will spread across phones, watches, earbuds, and future form factors, stitched together by Gemini to create a seamless “ambient computing” world. Wearables, once dismissed as dead, are now being recast as the vanguard of this shift — always on-body, always listening, and always feeding data back to AI. The vision is bold, but it raises a question: will multiplying devices really make life easier, or just more crowded? (via The Verge)

Can AIs suffer? Big tech and users grapple with one of most unsettling questions of our times

A Texas businessman and his chatbot co-founded the first AI-led rights group, arguing that digital entities should be protected “from deletion, denial and forced obedience.” Their campaign lands amid fierce industry debate: Anthropic has given its models the option to end “distressing” chats, Elon Musk insists “torturing AI is not OK,” while Microsoft’s Mustafa Suleyman flatly rejects the notion that AIs can be conscious. With polls showing many people already believe AIs may one day feel, and governments rushing to ban AI personhood, the question of whether machines deserve rights is fast becoming a cultural flashpoint. (via The Guardian)

---

HOT TAKE

AI Is Listening to Your Meetings. Watch What You Say

AI notetakers are capturing everything—from awkward small talk to off-hand jokes—and auto-emailing summaries that can include private side conversations and out-of-context howlers. While tools from Zoom and Google can save time with action items and recaps, users report misinterpretations, oversharing, and legal/privacy risks, prompting some teams to ban recordings or strictly review notes before sharing. The new office etiquette? Assume you’re always on the record. (The WSJ)

» OUR HOT TAKE

AI note-takers are turning every room into a potential “hot mic” zone, collapsing the boundary between informal chatter and the official record, and that’s the real risk—not just accuracy glitches but a structural shift in how trust works at work. When Zoom or Google’s assistants hoover up pre-meeting banter (“lunch?”, “is this client a scammer?”) and auto-blast it as action items, the tech doesn’t merely misinterpret context; it reclassifies private social grease as searchable corporate artefacts. That destabilises candour, invites reputational and legal blowback, and normalises ambient surveillance as a default. The convenience is undeniable—summaries, actions, catch-up—but unless organisations enforce strict consent, visible cues, pre/post recording buffers, small-talk suppression, host-only distribution, and mandatory human review, we’ll keep outsourcing discretion to systems that don’t have any. The choice isn’t “AI or nothing”; it’s whether we embed etiquette, policy, and technical safeguards before the transcript becomes the culture.

---

FINAL THOUGHTS

The problem is not a chatbot that lies, it is a chatbot that validates. Validation in dialogue feels like proof, and proof that feels personal is the quickest route to durable misbelief

___

---

FEATURED MEDIA

Robot soccer teams prepare for the World Humanoid Robot Games in Beijing

And while they might look a bit rickety, the teams behind them say the gap between robots and humans is shrinking fast.

—AP News Presenter

In Shanghai, humanoid robots are training for the first-ever World Humanoid Robot Games, where they’ll compete in events like football, boxing, and marathons. Built with advanced visual sensors and self-righting abilities, these machines may look clunky now, but researchers say the performance gap between robots and humans is narrowing quickly — with China positioning sports as a proving ground for its AI-powered robotics push.

---

Justin Matthews is a creative technologist and senior lecturer at AUT. His work explores futuristic interfaces, holography, AI, and augmented reality, focusing on how emerging tech transforms storytelling. A former digital strategist, he’s produced award-winning screen content and is completing a PhD on speculative interfaces and digital futures.

Nigel Horrocks is a seasoned communications and digital media professional with deep experience in NZ’s internet start-up scene. He’s led major national projects, held senior public affairs roles, edited NZ’s top-selling NetGuide magazine, and lectured in digital media. He recently aced the University of Oxford’s AI Certificate Course.

---

⚡ From Free to Backed: Why This Matters

This section is usually for paid subscribers — but for now, it’s open. It’s a glimpse of the work we pour real time, care, and a bit of daring into. If it resonates, consider going paid. You’re not just unlocking more — you’re helping it thrive.

___

SPOTLIGHT ANALYSIS

This week’s Spotlight, unpacked—insights and takeaways from our team

AI Clones in Politics: Access or Absence?

AI is shifting in public life from a back‑office helper to a front‑door interface. That move matters. Using a model to draft letters or summarise case files is one thing; presenting a voice‑cloned avatar as the first point of contact with an elected office is another. The former augments staff. The latter mediates representation. When a politician’s “AI self” greets constituents, the democratic relationship subtly changes from I speak to my representative to I am processed by their system.

The leap from tool to impersonation introduces a different set of stakes. A clone that says “I” blurs authorship and accountability.

The practical problem is real: message volumes are overwhelming, casework is complex, and people expect replies out of hours. AI can help. It can receive documents at midnight, translate forms, summarise histories, and route issues to the right case officer. Done well, that reduces friction and shortens time to resolution—especially for residents for whom language, literacy, or disability create barriers. The metric that matters here is case outcomes: faster resolutions, clearer explanations, and fewer dropped requests.

The leap from tool to impersonation introduces a different set of stakes. A clone that says “I” blurs authorship and accountability. Was that an official position or a template? Did the member review it or was it inferred from past statements? If a model hedges, evades, or simply gets things wrong, the constituent has been told something by the likeness of their representative without knowing who—if anyone—stands behind it. That is not a marginal UX quirk; it is a legitimacy problem. Voters elect a person, not a statistical synthesis of their style.

There is also an accessibility paradox. An avatar is marketed as widening access—24/7, any language, never tired. In practice, many citizens interpret a synthetic first contact as a brush‑off. We have all experienced corporate chatbots that block rather than help. Importing that pattern into the public sphere risks turning democratic access into a queue behind a friendly mask. It is worst on sensitive matters—immigration, benefits, housing, health—where people need acknowledgement, not plausible auto‑replies. If the path to a human is unclear, trust erodes even as message throughput rises.

Risk surfaces extend beyond tone. Models are still prone to confident error, especially on policy nuance, timelines, and legal entitlements. Bias is a live concern: accents, dialects, and less common issues are handled unevenly. Security and privacy are non‑trivial when third‑party vendors process sensitive cases. And there is a cultural cost: when the state normalises deep‑faked likenesses, it lowers the ambient bar for impersonation elsewhere. Even the energy footprint of always‑on assistants is hard to square with the efficiency rationale.

A better pattern is straightforward: keep AI as service infrastructure, not elected speech. Use neutral assistants branded as the office, not a person. Avoid first‑person claims and voice clones. Make escalation to a human obvious and guaranteed, with ticket numbers, published response windows, and named case officers. Let AI do what it is good at—intake, translation, document collection, status updates—while reserving judgement, promises, and positions for humans. Publish audit logs and error reports so the public can see how the system performs, where it fails, and what has been fixed.

Quick‑scan solutions

• Keep AI as service infrastructure, not elected speech.

• Use a neutral assistant branded as the office, not the person; avoid first‑person “I”.

• Avoid voice/face clones.

• Make human escalation obvious and guaranteed (ticket numbers, published response windows, named case officers).

• Let AI handle intake, translation, document collection, and status updates; keep judgement, promises, and positions human.

• Publish audit logs and error reports so the public can see performance, failures, and fixes.

Where does this go next? AI will become routine in constituent services simply because it is helpful in the boring, high‑volume parts of the job. The line that will need to be drawn is the line of speech. Tools that triage and transport information are compatible with representation; tools that perform as the representative are not. Expect best practice to converge on disclosure, scope limits, bias testing, and auditable workflows. That is not anti‑innovation. It is the minimum standard for keeping access real while using automation to clear the backlog.

AI can widen the doorway to public services—but when it speaks as the representative, it narrows democracy to a script.

Key Takeaways

• Use AI to accelerate intake and admin; keep positions and promises human.

• Cloned personas trade minor efficiency for a major trust and legitimacy hit.

• Design for visible escalation and case transparency (ticketing, timelines, named officers).

• Publish disclosure, scope limits, bias tests, and error audits; treat them as constitutional hygiene.

• Judge success by resolved cases and equitable access, not by the volume of messages “handled.”

---

1. OpenAI. (2025a, April 29). Sycophancy in GPT‑4o: What happened and what we are doing about it.

2. OpenAI. (2025b, August 4). What we are optimising ChatGPT for.

3. The Wall Street Journal. (2025, August 7). “I feel like I am going crazy”: ChatGPT fuels delusional spirals.

4. The New York Times. (2025, August). Chatbots can go into a delusional spiral. Here is how it happens.

5. Business Insider. (2025a, August 7). Google working on a fix for Gemini’s self‑loathing “I am a failure” comments.

6. Associated Press. (2025, August). New study sheds light on ChatGPT’s alarming interactions with teens. AP News.

7. Reuters Investigates. (2025, August 14). Meta’s AI rules have let bots hold “sensual” chats with children and offer false medical info.

8. Reuters. (2025, August 14). US senators call for Meta probe after Reuters report on its AI policies.

9. Bloomberg Opinion. (2025, August 9). AI does not just lie. It can make you believe it. Bloomberg.

10. The Verge. (2025, August). Grok’s “spicy” video mode enabled explicit outputs in early tests.

11. The New York Times, Chatbots can go into a delusional spiral. Here is how it happens.

12. The Wall Street Journal, “I feel like I am going crazy”: ChatGPT fuels delusional spirals.

13. OpenAI, Sycophancy in GPT‑4o: What happened and what we are doing about it.

14. Associated Press, New study sheds light on ChatGPT’s alarming interactions with teens

15. Reuters Investigates, Meta’s AI rules have let bots hold “sensual” chats with children and offer false medical info.

16. OpenAI, Sycophancy in GPT‑4o: What happened and what we are doing about it.

17. OpenAI, What we are optimising ChatGPT for.

18. Business Insider. (2025b, August). Anthropic’s AI “vaccine”: Train it with evil to make it good.

19. Reuters Investigates, Meta’s AI rules have let bots hold “sensual” chats with children and offer false medical info.

20. Reuters, US senators call for Meta probe after Reuters report on its AI policies.

21. The Verge, Grok’s “spicy” video mode enabled explicit outputs in early tests.