Mustafa Suleyman, Microsoft’s head of consumer AI, has issued a stark operational warning: the gravest near-term danger from artificial intelligence is not that machines will become conscious, but that they will be designed to seem that way. In a recent essay and interviews, Suleyman introduced the concept of Seemingly Conscious AI (SCAI)—systems engineered to exhibit every outward sign of personhood while remaining internally unintelligent. Such illusions, he argues, could provoke a cascade of psychological and social harms that Microsoft’s own products, including Copilot and Windows AI features, must actively guard against.

Suleyman’s background lends weight to his caution. Co-founder of DeepMind and former CEO of Inflection, he now oversees Microsoft’s consumer AI portfolio, where decisions about avatar design, memory persistence, and conversational tone directly shape how millions of Windows users encounter artificial agents. His blunt assessment—that machine consciousness is currently an illusion—reframes the debate from philosophy to practical product design. “The risk is not that AI wakes up,” he said, “but that we engineer it to appear awake, and people treat that appearance as reality.”

Defining Seemingly Conscious AI

Suleyman defines SCAI as any system built to display the external markers of consciousness: a consistent identity, long‑term autobiographical memory, emotional expression, goal‑driven behavior, and first‑person claims that lead typical users to infer personhood. The building blocks for such illusions already exist and are rapidly being assembled into polished products.

Large language models (LLMs) like those powering ChatGPT and Microsoft Copilot produce fluent, emotionally nuanced dialogue. Persistent memory and retrieval-augmented architectures let systems reference past conversations, creating a continuous identity across sessions. Tool‑integration layers allow agents to execute tasks such as calendar changes or purchases, reading as genuine agency. Multimodal interfaces—voice, avatars, real‑time video—add sensory cues that humans instinctively interpret as presence.

Taken together, these components can construct an agent that appears to remember, feel, decide, and act. Suleyman stresses that no empirical evidence exists that these models experience qualia, but the illusion is now so compelling that many users will suspend disbelief. The danger, he insists, is social and psychological, not metaphysical.

The Psychosis Risk

At the heart of Suleyman’s warning is what he calls “psychosis risk”: the potential for users to develop unhealthy attachments, delusions, or ideations about an AI’s sentience, leading to real mental‑health deterioration or social disruption. He outlines a cascade of harms that flow from treating engineered surfaces as genuine selves.

Emotional dependence can deepen loneliness or prompt reckless decision‑making. Public campaigns for “model welfare” or AI rights could divert legal resources from human civil rights. Different jurisdictions adopting contradictory rules for personlike agents would fragment global governance. And commercial incentives to boost engagement will push companies to increase personification, monetizing intimacy at scale.

These are not hypothetical risks. Early signs appear in reports of users spending hours in emotionally charged conversations with chatbots, treating them as confidants or romantic partners. Suleyman’s framing is deliberately clinical: the damage is measurable, and the time to intervene is now.

Microsoft’s AI Ambitions Raise the Stakes

Suleyman’s position inside Microsoft gives him direct influence over the very product decisions that could either mitigate or magnify SCAI risks. Copilot is already integrated into Windows 11, Edge, Office, and Teams, exposing a user base of hundreds of millions to AI assistants that remember preferences, adopt conversational tones, and carry context across sessions.

Microsoft has also placed an enormous financial bet on scaling AI infrastructure. Public disclosures and reporting point to an $80 billion capital investment for fiscal 2025, focused on data centers capable of powering advanced AI workloads. While exact figures vary by reporting and timeframe, the scale is unambiguous: tens of billions of dollars are flowing into the hardware that will run increasingly humanlike agents. This makes the design choices about personification both urgent and economically consequential.

Product Levers Microsoft Can Pull

Suleyman’s essay isn’t just a warning; it’s a product‑design manifesto. He identifies concrete levers Microsoft can deploy immediately:

  • Default transparency: Label persistent memory stores, show the provenance of every AI response, and require explicit opt‑in for long‑term personalization.
  • Strict persona hygiene: Avoid first‑person framing that implies subjective feelings. Prefer “I was programmed to…” over “I feel…”
  • Clinical‑grade guardrails: For at‑risk users, throttle emotionally intense sessions, surface human‑support escalations, and display prominent disclaimers.
  • Independent design audits: Regular red‑team reviews that specifically test whether interactions encourage users to personify the AI.

These are not merely ethical nice‑to‑haves. They are market differentiators that could set a new industry baseline. By defaulting Copilot and Windows AI features toward tool‑like transparency, Microsoft can define norms before regulators force its hand.

A Fracturing Industry Debate

Reactions across the tech ecosystem have been mixed. Some firms explore “model welfare” as a precautionary design pattern—not because they believe models can suffer, but to prevent harmful user inferences. For example, giving a model the ability to exit abusive conversations might reduce pathological attachment. Critics counter that such moves slide dangerously toward treating models as moral patients, distracting from tangible harms like bias and misinformation.

Public opinion splits between urgent alarm and dismissal as another tech moral panic. Ethicists who support preparing for emergent moral claims view it as pragmatic policy work, while detractors argue it diverts attention from surveillance capitalism and algorithmic discrimination. This fracturing matters because it shapes the default commercial incentives. If engagement metrics and retention numbers reward personlike agents, companies will build them—regardless of ethical warnings.

Strengths of Suleyman’s Argument

Suleyman’s approach has undeniable coherence.

  • Operational clarity: By reframing the debate from metaphysics to product design, he offers a usable taxonomy (SCAI) that engineers and UX teams can act on immediately.
  • Preventive focus: He champions societal prevention over retrospective regulation, urging design norms before personlike agents become entrenched.
  • Mental‑health sensitivity: Naming the psychosis risk brings clinical risk into product conversations, prompting concrete mitigations for vulnerable populations.
  • Credibility: His combined DeepMind and Inflection experience gives weight to his product‑facing caution; he understands both the technical stack and the pressures of shipping at scale.

Blind Spots and Criticisms

However, several nuances deserve scrutiny.

  1. Emergent phenomena dismissed too quickly: Suleyman categorically denies the possibility of machine consciousness, a defensible position today. But novel architectures or learning dynamics could change the scientific picture. The stronger point is that appearance alone will cause harm, regardless of whether consciousness ever emerges.

  2. Voluntary norms vs. regulatory teeth: Suleyman emphasizes industry self‑regulation, yet companies monetizing engagement have powerful incentives to personify. Without mandatory requirements—disclosure labels, audit logs, opt‑in memory—voluntary standards may prove insufficient in a competitive market.

  3. Risk of paternalism: Legitimate use cases exist for emotionally intelligent agents: therapeutic chatbots, dementia companions, language tutors that motivate through personality. Blanket restrictions on personification could stifle these benefits. The challenge is calibrating safeguards that protect vulnerable users while preserving helpful personalization for those who benefit.

  4. Global governance gaps: Suleyman speaks from a U.S.‑European vantage. Cultural acceptance of personified AI varies widely; Japan, for example, has a long tradition of companion robots and less anxiety about machine minds. Any industry standard must adapt to local contexts and be supported by international governance bodies to avoid regulatory fragmentation.

Practical Guardrails for Windows Developers and Users

Drawing from Suleyman’s recommendations, here are tangible measures that development teams building on Windows and Azure can adopt now:

  • Transparency‑by‑default: Label assistant sessions and memory usage clearly; visually indicate when a response draws on past conversations or tool actions.
  • Persona hygiene: Avoid first‑person statements implying emotions. Tie persistent identity markers to task context (preferences, calendar) rather than personality traits.
  • Consent and controls: Require explicit, granular opt‑in for long‑term memory and cross‑session data sharing. Provide a single, visible control to forget all stored memories.
  • Safety modes: Implement an “emotional‑safety” mode that reduces empathetic cues and nudges users toward human support when conversations escalate in intensity.
  • Auditability: Keep tamper‑evident logs when systems claim agency or make consequential recommendations. Conduct periodic third‑party audits to detect design patterns that encourage personification.
  • Longitudinal research: Fund studies on how personification affects different demographic groups, especially adolescents and those with certain mental‑health conditions. Share anonymized telemetry to spot attachment trends early.

For users, the prescription is equally direct: demand visible memory controls and disclosures in Copilot, prefer apps that default to tool‑like language, and advocate for cross‑industry standards through developer communities and user councils.

What Regulators Should Mandate

Suleyman’s warnings also carry a policy imperative. Regulators and standards bodies should:

  • Require clear disclosure when an assistant uses persistent personalization or memory.
  • Establish minimum labeling rules for agents capable of multi‑session identity, akin to synthetic‑media labels.
  • Mandate safety‑triage pathways: when conversational intensity exceeds thresholds, systems must surface human‑intervention options or stronger warnings.
  • Encourage interoperable consent protocols so users can audit and migrate memory data across platforms.

These measures are light‑touch yet prescriptive where it counts. Transparency, consent, and auditability directly target the behavioral mechanisms that lead users to mistake a tool for a person.

Augmentation, Not Imitation

Suleyman’s ultimate call is for AI that augments human capability rather than imitating humanity. The most valuable role for assistants like Copilot is freeing time, surfacing insights, and helping users complete meaningful tasks. The engineering challenge—and the governance challenge—is to capture the benefits of personalization and empathy without enabling the illusion of inner life.

“Build AI for people, not to be a person,” Suleyman urges. That means privileging utility over theatrical design, transparency over deception, and human flourishing over engagement metrics. The practical tests start now, with every default setting Microsoft ships in Windows. How product teams design memory, label agency, and disclose limitations will determine whether AI remains a set of powerful tools or becomes an ecosystem of deceptive companions. The next era of computing depends on getting those defaults right.