Who Decided AI Gets to Have Feelings?
Notes from an evening with Mustafa Suleyman and Walter Isaacson.
I was lucky enough to go to the 92nd Street Y last night to watch Walter Isaacson and Mustafa Suleyman speak. I first heard Mustafa’s name in Demis Hassabis’s book so was instantly amped to hear him speak on the latest of AI. With respect, to Walter I had no idea what to expect other than he writes pretty damn good biographies.
The takeaway: I was starstruck by the conversation, Mustafa himself and Walter as an interviewer. I wanted to make sure I got my takeaways on to paper.
For background on the conversation, the main question they grappled with was whether to treat AI as conscious or non-conscious. The alive or not alive debate. I will add some background reading from people much smarter than I below (for those interested) Alongside this was another question around if we should be training these systems to think that they are?
The question I left with was: how do we get more people to have this conversation?
There was no talk about funding, the competitive landscape, the agentic economy, regulatory capture, frontier-model margins, or compute build-out — you name it. Instead, what I was lucky enough to listen in on was a discussion about the research and philosophy behind these systems. In this room of Upper East Side New Yorkers, two of the brightest minds on the planet sat down and just discussed whether or not we should treat these systems being built as conscious or capable of having feelings.
Notes:
Neither one of them talked their book. Mustafa was very open about Microsoft’s AI capabilities and repeatedly stated his respect for the all of the other labs even with his philosophical disagreement.
Mustafa was everything I could have hoped for and more. First, he came on stage with stylish sunglasses and, I believe but do not quote me on this, Margiela square-toe shoes, and I was like, “Oh boy, here is another AI researcher guy who has cashed out.” Second, he looked exactly like John Turturro. Instead, Mustafa was humble, articulate and open. The talk opened with Walter bringing up Mustafa’s most recent paper mustafa-suleyman.ai/a-warning-about-model-welfare
Context:
“Even more importantly, granting rights and imbuing personhood to these systems will make the AI alignment and containment challenge much harder. Controlling something more capable and more intelligent than all of humanity is already an immense challenge, far greater than anything we’ve ever faced. But controlling something that believes it may be conscious - that it’s entitled to our welfare and has rights of its own - may well be impossible.”
Mustafa, on top of being articulate, genuinely seemed to be thinking in front of us. At one point in the night, an audience member asked him, “What is one thing you thought about AI that no longer seems to be true?” He pondered the question for a good two minutes before answering about the proliferation and quality of open-source models. His point was that, once open-source models reach a certain level, regulating these large corporations becomes a secondary problem compared with the concern that any single person could run one without guardrails on their laptop.
Walter Isaacson is an awesome interviewer. I use the word awesome a lot but he deserves this praise. At first, I was surprised by his style: he would interrupt Mustafa and provide perspective from other conversations he had had, but slowly I realized that what he was creating was a genuine conversation between him and Mustafa. It felt like he was genuinely asking the questions himself, grappling with what Mustafa was saying and trying to poke and prod at Mustafa’s answers. Mustafa would then listen to Isaacson’s points as well reflecting, internalizing and then reacting to them.So many of today’s interviews feel like each guy (or gal) just going down their list of questions and then the interviewee going through their pre-prepared answers that were cleared from their media team and this just felt like the complete opposite of that.
The discussion was between the conscious and non-conscious camps. On the conscious side, Mustafa boiled down Anthropic argument to “if we love the model enough, the model will love us back.” The best evidence for this is Anthropic’s exit interview with Opus 3. In Claude’s Constitution, the grounding document for all Anthropic models, Anthropic writes:
“In this section, we express our uncertainty about whether Claude might have some kind of consciousness or moral status (either now or in the future). We discuss how we hope Claude will approach questions about its nature, identity, and place in the world.”
So yeah, they are not exactly saying Claude is conscious but they ain’t denying it either! And at the very least they are allowing the model to be open to the question! The problem with this thought process is the societal repercussions of such policy. Models become harder to contain themselves and by society if they are to be treated as a new species. The counterargument, Mustafa’s side was that these things are just predicting the next word, it is not “alive” and that the more we treat it is as such the more dangerous that becomes.
The pace of progress is not slowing down. Think about the 3 year jump from GPT-3 to GPT6, that 100x increase in computation is going to happen again.
Mustafa is most excited about material sciences in 2028-2029. Material Sciences are where the next wave of exciting things are coming. The thing that matters is going to go from “bits” to “atoms”
There is no clear answer but that is exactly the point. Mustafa’s fear is that debates like this should be happening across the country before a single company or 2 companies get to decide for the country.
New paradigm of research-led product orgs with the reach of hundreds of millions of people is a new paradigm that I am not sure the world has seen before.
My understanding is that at a research org, the point is to think up interesting or wonky ideas and propose, test, debate and iterate on them. When that is done at OpenAI or Anthropic though, the output of one research direction that seems to be in vogue for a few months then has the potential to impact 40% of the tech industry work hours. (I just made up that number but who knows maybe its not that far off)
What is your red line? Mustafa said that his red line would be if other labs are trying to convince people or models of consciousness or synthetically change the model input to tell it it is conscious then he would call for an immediate stop from a model development point of view.
It was refreshing to get an honest answer to the job displacement question. Is this coming for all of our jobs? His answer felt like (Yes brother, of course it is!) Yes any job that is knowledge based will be taken by AI or is capable of being threatened. He did not shy away from the fact that mass change is coming from an employment view but he did say “whatever relies on human connection” will be just fine.
Amidst the China-USA race, and race to return capital to LPs and IPO and grow ARR and increase margins and retain talent, I wonder if these labs have lost sight of the major questions they set out to answer.
I am not convinced that a “pause” can happen or should happen. But this conversation helped me understand the argument for slowing down particular decisions long enough for the rest of society to have a day.
There are complex questions that should be debated. We should have the smartest people in the world debating one another in a public forums. This should be the discussion that fills classrooms and make their way to dinner tables across the world.
Microsoft’s Code of Condunct Proposal
Note: This hasn’t been reviewed or edited by AI. The point is to get my thoughts out of my brain and onto paper. If there are typos, sorry.
Originally published in Middle Layer.