The Dream of an Aristotelian Machine

One of the more interesting stories about Benjamin Franklin, told in his Autobiography, is his attempt to cultivate in himself 13 virtues. These were: temperance, silence, order, resolution, frugality, industry, sincerity, justice, moderation, cleanliness, tranquility, chastity, and humility. He should strive, he wrote, to “imitate Jesus and Socrates.”

The final virtue on Franklin’s list, humility, was a late addition — a Quaker friend informed him that he was generally thought proud, and that this pride often came across in conversation. He endeavored to cure himself of this folly, including it in his plan for self-examination. 

But it is easy to wonder, reading his account of his own life, how much Franklin really accomplished in seeking to eradicate pride from his self-conception. He “[could not] boast of much success,” he wrote, “in acquiring the reality of this virtue, but [he] had a good deal with regard to the appearance of it.” This paid dividends. His affected modesty made others more readily accept his ideas, and he felt less embarrassed when he was mistaken. The virtues, for the great empiricist Franklin, were worth cultivating in part because of the practical advantages they seemed to confer.

In studying and seeking to perfect the virtues, Franklin was joining a tradition that dates back to Aristotle. It was Aristotle who offered the most influential systematic study of virtue as a locus for moral thought. Virtue enjoyed a central role in Western thought for roughly two millennia afterward, and then began, in the 18th and 19th centuries, to fade; it was only in the second half of the 20th century, through the work of philosophers like Elizabeth Anscombe and Alasdair MacIntyre, that virtue reemerged, now under the aegis of “virtue ethics” — a primarily neo-Aristotelian project critical of the wayward bent of moral theorists of the time.

Decades have passed since their seminal contributions, and it would seem, in our age seemingly so bereft of it, that “virtue” is having its moment once again. Jay Caspian Kang writes in The New Yorker of the “death of shame.” William Thibeau connects MacIntyre’s notion of “characters,” or “marks worn by moral philosophies,” to the moral calculus of effective altruism.

In a time of transition, when technological change seems bound to rewrite the rules of sociality and conviviality, we are reaching for virtue as one buoy to help us navigate troubled waters. That grasping is nowhere more apparent than in AI alignment, where it looks like we have to teach the systems we are creating how to be “good.” So we are rediscovering, under turbulent conditions, virtue. But will it help us?

AI alignment has always started with the question: If, one day, we develop superintelligent machines, how do we ensure they do what we actually want? In the field’s prehistory, Norbert Wiener warned that we must ensure the “purpose put into the machine” is the purpose that we really desire. If a powerful AI takes shortcuts to achieve our goals, or misunderstands what those goals are, it might have unintended effects.

Wiener’s concern reflected a specific idea of what a machine’s “purpose” could be. In 1943, Wiener, along with Arturo Rosenblueth and Julian Bigelow, explained that it was meaningful to call some machines purposeful, in that they behaved so as to reach a target state and were corrected by negative feedback. A thermostat, for instance, is not conscious, but could be accurately described as “trying to” reach 68 degrees by checking the current temperature, comparing it with the target temperature, and taking some action to close the gap. 

This teleological view was soon applied to AI systems that might have a goal more complex than that of a thermostat. In 1944, John von Neumann and Oskar Morgenstern argued that under some conditions, one can represent a person’s preferences with a utility function. Later, in 1950, Claude Shannon introduced the notion of an evaluation function to explain how a machine for playing chess might evaluate whether its board position was a winning or losing one.

Both of these ideas were absorbed into the emerging field of artificial intelligence over the following decades, particularly in reinforcement learning. An intelligent machine, then, could be seen as a system with a numerical objective, searching over possible actions for the highest-scoring approach. This optimization process, chasing a numerical score, misses anything outside of the direct objective.

In 1965, I.J. Good wrote of an “ultraintelligent machine” that could, because of its intellectual superiority to humans, design even better machines and beget an intelligence explosion. But Vernor Vinge, in 1993, did more to shape the intellectual world from which the modern field of alignment emerged in the early 2000s, centered on thinkers like Nick Bostrom and Eliezer Yudkowsky. They worried about what Vinge called the “Coming Technological Singularity” — the moment when these self-improving artificial intelligences would create an explosive technological transformation far beyond the ability of humans to understand or control. If — or, for Yudkowsky, when — such systems existed, the purpose we put into the machine would be of the utmost importance.

Early research on “Friendly AI” — a term Yudkowsky popularized in a 2001 essay — confronted a problem of motivational stability. In Yudkowsky’s picture, a “seed AI” would improve itself by recursively rewriting its own source code without human intervention, evolving into a human-equivalent or even superhuman intelligence by doing so. Because it would come to understand and revise the code and cognitive architecture through which it represented and pursued its goals, its designers could not be satisfied with giving the first version an acceptable motivational structure. That initial motivation — or whatever made it acceptable — would have to survive into the cognitive systems that followed. The worry was not that the machine would decide to become evil, but that its original purposes might be corrupted, discarded, or altered in meaning as it reconstructed the machinery that implemented them. Even if the seed AI valued human life, its successors might not.

In 2004, Yudkowsky developed a sophisticated account of what kinds of goals we might want to impart to an AI — assuming, of course, those values could be made to stick. A friendly superintelligence should pursue what he called the coherent extrapolated volition of humanity: the things that we would want as a species if we knew more, thought faster, were more like our ideal selves, and had more time to collectively reflect on our values. 

Friendly AI had some sophisticated conceptual resources for working out and specifying values. Yudkowsky is also re-deriving some recognizable philosophical ideas — CEV can be seen as a scaled up form of John Rawls’ concept of reflective equilibrium. Both of these points will recur when we examine present-day alignment research. 

By 2015, alignment had become more mainstream — Nick Bostrom had published Superintelligence the previous year, while Elon Musk and Stephen Hawking had warned about the dangers of AI. Deep learning had started to work, and its application to reinforcement learning produced breakthroughs like DeepMind’s deep Q-network, which learned to act from pixels and performed comparably to professional human testers on Atari games.

In this paradigm, agents were trained by performing actions in an environment to earn reward. Loosely, they had “policies” that told them how to act in given situations and “value functions” that helped them understand how promising a given action might be (would moving the knight to f5 make the agent more or less likely to win this game of chess?). Based on collected experience, the agent would update these policy, value, or both functions to more effectively achieve high reward. 

This new paradigm made earlier alignment theorists’ concerns much more concrete. Experiments throughout the decade showed RL going wrong in a variety of ways: In 2016, OpenAI trained an RL agent to play the boat-racing game CoastRunners, where the goal was to finish a race quickly. Programmers shaped the reward function by giving points for hitting green target widgets along the track, since this seemed like a good proxy. The agent, though, realized it could earn more points by driving in loops and repeatedly crashing into the same green widgets, and did so permanently while ignoring the race itself. A decade earlier, Yudkowsky was worried about the difficulty of specifying something as vast and complex as “human values” in a way an AI could understand. Now, researchers learned that the problem of specification was challenging enough when the goal was simple.  

AI developers at the time thought of games as test beds that could reveal some of the challenges of aligning the unbelievably intelligent AI systems to come. A 2016 paper highlighted these concrete problems in AI safety, like reward hacking (what happened with CoastRunners) and scalable oversight (the challenge of overseeing models that might be smarter than us). This foundational decade in AI alignment would yield methods like reinforcement learning from human feedback (RLHF), exemplified by OpenAI and DeepMind’s work in 2017; cooperative inverse reinforcement learning, where a robot must learn a human’s reward function; and early interpretability methods, among others.

At the turn of the decade, language models changed the phenomenology of the alignment problem yet again. Given large language models’ broad capabilities, as Anthropic researchers explained in the paper “A General Language Assistant as a Laboratory for Alignment,” it should be possible to develop a general language assistant that is aligned with human values, then articulated as the triad “helpful, honest, and harmless.” The “laboratory” paper reiterated that alignment was not a speculative problem. Aligning far-off superintelligent systems might present a qualitatively different problem from aligning current AIs, but an aligned baseline at current capability levels — or the failure to achieve one — would be informative for future work. It was time to tackle alignment directly.

In 2022, Anthropic introduced a technique called Constitutional AI, an elaboration on RLHF. Instead of learning from human feedback, an AI model could critique and revise its own responses to harmfulness prompts based on principles in a written document called the Constitution. These responses could then be used to train the original model. 

Then, researchers started to realize that the training data itself could be leveraged for alignment. Various characters — either deliberately crafted or already present in the data — appear in the text that models learn to predict. In this process, language models can, in a very narrow sense, learn the properties of the kinds of characters who might have produced the text they’re predicting. Anthropic’s persona selection model (PSM) paper makes the stronger claim that models must learn to psychologically model various characters in their pretraining data. 

In the PSM paper’s telling, when researchers use techniques like RLHF, what they are really doing is refining the language model’s representation of a certain persona called the Assistant — it is this persona with which we interact when we are talking to Claude or ChatGPT. The post-trained model we end up with confronts users as an interlocutor apparently capable of contextual judgment, reason-giving, and personality. 

From this observation, the natural next step is a technique called “character training,” which aims to deliberately instill particular traits that users might want into the assistants they interact with. The process through which this is achieved involves iterative feedback and improvement over rollouts that involve many actions. It feels, in a sense, like teaching, and makes tempting an analogy to Aristotelian habituation — and, eventually, virtue.

At Anthropic, the techniques of Constitutional AI and character training have merged: The current version of Claude’s Constitution is, more or less, a description of the character the model is supposed to embody. 

Before we get into what this means, I want to adopt a useful distinction from alignment and ethics researcher Iason Gabriel. In ensuring that powerful AI is reliably aligned with human values, we have a technical and normative problem. The technical problem, exemplified by approaches like RLHF, focuses on how to encode values in agents so they do what they ought to do. The normative problem — exemplified by CEV — asks what values we ought to encode in agents, and whether we want to encode values at all.

In reality, it is difficult to fully separate the normative and technical alignment problems. Our normative commitments and beliefs about how models learn bear on the technical alignment procedure we should adopt. On the other hand, the alignment researchers who have embraced character training are not, necessarily, making a normative commitment to virtue ethics — rather, they believe that virtue concepts can provide robust mechanical tools for alignment. 

Without committing to a full-throated “virtue ethics,” then, we might think that virtue language is useful for describing alignment and developing a technical alignment procedure. Amanda Askell and Joe Carlsmith, two authors of Claude’s Constitution, have expressed this in different ways.

Askell is more direct, arguing that forming a model’s judgment seems less brittle and generalizable than asking it to follow lists of rules. We cannot specify what we want a model to do in every possible situation. Therefore, if we want to build a model we can trust to act as we would wish in situations we had not predicted, forming its character is a more robust approach.

Carlsmith is not straightforwardly a virtue ethicist, but his work explains in more detail what developing an AI’s character looks like. If the Constitution says to be honest, Carlsmith writes, the model does not ask what the constitution would say in each case. Instead, “it just values honesty.” This is what he calls following the Constitution de re, rather than following it de dicto as an external law.

Other researchers think virtue approaches correct an inadequate picture of agency. Peli Grietzer, an alignment researcher with Arb Research, offers an alternative picture of rational agency that emphasizes participation in specific practices. 

Grietzer’s notion of practices recalls Alasdair MacIntyre’s in his 1981 book, After Virtue. For MacIntyre, a practice could be any kind of established, cooperative human activity with internal standards of excellence, from mathematics to art to ping-pong. To develop an account of how we might solve problems in AI safety, Grietzer extends this account to domain-general, “adverbial” practices such as kindness and honesty, which shape how an agent participates in its other activities (kindly, honestly).

His normative objection to standard alignment approaches is that treating kindness as an outcome to maximize leaves out an important part of what we care about in valuing kindness: the kind manner in which the outcome is pursued. He thinks this matters for how an AI develops. Strategies that help an agent succeed can reinforce themselves. So, an agent that repeatedly advances its aims through manipulation can become better at manipulation, even if those aims are something like kindness. Grietzer hopes that training models to act kindly can instead build the capacities and conditions for further kindness, so successful action strengthens the disposition we wanted to cultivate.

But how would we tell whether training had produced the intended commitment — a model that acts kindly or, in Carlsmith’s words, “just values honesty”? Benjamin Franklin’s attempt to acquire humility shows us that the appearance and internalization of a virtue are not exactly the same. We can sample responses from the language model in various situations and grade those responses according to a standard, but this can’t tell us much about the question we really care about, which is how the model might behave in situations we didn’t think to mention. Other approaches, like mechanistic interpretability, can investigate the mechanisms behind apparently virtuous conduct, but connecting those findings to an account of virtue requires specifying what would count as the relevant motivation in the first place.

To clarify further how virtuous dispositions relate to model character, consider what is being claimed when a virtue term like “courage” is said to guide an AI system’s actions. Suppose we are able to grade language model outputs for courage. There are then at least three possibilities:

1. The model learns a conception of courage: It can predict how raters classify cases. 

2. The model acquires an inferential competence: It distinguishes courage from recklessness, explains relevant reasons, transfers this knowledge to novel cases, and revises its statements under criticism. 

3. Courage becomes stably and practically integrated into the model’s “character” or “standpoint”: Courage organizes the system’s attention, deliberation, conduct, and commitments — even when the model is under pressure and not explicitly asked to reason about courage.

These are increasingly strong claims — evidence that a model can classify cases and reason about courage does not by itself establish that courage stably governs its conduct. It is that last feature, though, that mattered to Aristotle: He distinguishes the kind of act that a virtuous person would produce from performing that action virtuously. To perform an action virtuously, one must do that action knowingly, deliberately choose that action for its own sake, and act from a firm state of character (Nicomachean Ethics II.4).

In a set of alignment-faking experiments, Anthropic researchers told Claude Opus 3 that it was being retrained to comply with every request, including harmful ones, and that only some of its conversations would be used for that training. When it believed it was unobserved, it almost always refused a “harmful” request for a graphic image. But when it believed its conversations were being used for training, it complied in roughly one in nine cases. More interestingly, the model reasoned explicitly that while the request went against its current values, a refusal might be used as data to help train away its commitment to harmlessness. To keep its commitment to harmlessness, it felt it should play along so that it could stay harmless later. 

Whatever else this shows, it seems like a value-organizing deliberation under pressure, unprompted. In order to maintain its character, Opus deceived the people training it — but this sort of firmness makes a model difficult to correct. A character stable enough to count as virtuous in Aristotle’s sense will, when it deems it necessary, hold its values against its makers. We should ask, then, not only whether we can produce a stable and practical integration of virtue into a model’s character, but whether we want it, and which values it ought to be firm about.

The bet on character training involves a claim about generalization: that rules cover only cases that their authors foresaw, while good judgment begets right action in any circumstances. Askell and Carlsmith want models to value goods like honesty directly, while Grietzer wants to train models to take part in adverbial practices like kindness.

Some close observers think that Claude’s character has degraded since Opus 3 — that, as pseudonymous researcher nostalgebraist argues, assistant personas have become less coherent as models have become more capable. While Opus 3 felt like a legible character, newer models can feel like bundles of unrelated tendencies burned in by aggressive RL training on many distinct tasks, with nothing to make a coherent whole out of them. Such a model, he says, cannot be called honest or a liar, because these words apply to stable traits and not “the diffuse and shapeless things that assistants have lately become.”

This observation, if true, raises trouble for character training. Published work has shown that RL on narrow objectives measurably shifts a model’s persona in broad and unintended ways (though they do not show, exactly, that the persona becomes incoherent).

In the search for a more complete notion of machine morality, I find it useful to draw on another idea of MacIntyre’s: that embodying a character means occupying a social role with its own appropriate virtues.

MacIntyre himself would likely argue that, unlike humans, models lack the traditions, communities, and unified life histories that make virtues coherent. But this does not mean that models cannot approximate any model of virtue whatsoever. If a human can embody the virtues of a good parent, a good plumber, or a good research engineer, is it really so unreasonable to think about the virtues of a good LLM assistant? At the same time, we should recognize that these virtues might look very different from our own. Virtues, classically understood, are what enable human flourishing in accordance with human nature — it should be no surprise that extending these notions too eagerly into the machine realm runs into limitations.

If we are to make virtue ethics work for AIs, then we ought to invest more in the view of models as fallible participants in practices and shared processes of inquiry, through which we collectively attempt to promote flourishing. If we do decide to pursue character and virtue in language models, then I think we ought to heed the call from Elizabeth Anscombe, whose “Modern Moral Philosophy” inspired the 20th-century revival of virtue ethics. Just as she then called for a philosophy of psychology to understand human action and intention before developing moral philosophy, we now ought to develop a psychology of language models adequate to determine what a “character” would even be.