Erik Nygren
6.868 Final Project
May 16, 1996

What is Morality?

If we lived in a world where our actions had no consequences, there could be nothing wrong with anything we might do. However, this is not the case. We are social animals, and the actions we take -- the things we do and the things we don't do -- have consequences on our environments and on the others around us. As a result, we need to be able to govern our behavior in the near term so as not to injure ourselves or our community in the long term. This system of controlling our actions and our thoughts in order to operate in a community is what we often refer to as morality.

What sorts of things do we commonly attribute to morality? Here are some common examples:

We shouldn't steal things.

Old ladies should be helped across the street.

Killing or injuring people is wrong.

We should give time and money to charities.

We shouldn't tell lies.

Looking at this list, we can see that there are two categories: things we should do, and things we shouldn't do. Looking closer, we will see that things we should do are inspired by empathy and a desire to help others. The things we shouldn't do are actions and thoughts that we censor and suppress. I will argue that what we call morality is actually two separate, although occasionally interacting, agencies. The empathic response agency inspires us to take action to help others while the moral restraint agency censors and suppresses "immoral" actions and the thoughts which might lead to them. Together these make up the Society of Morality

Why are there two agencies? Although they occasionally interact, both perform significantly different functions. The moral restraint agency often needs to take immediate action to keep us from doing things we would regret later or which would have unfortunate consequences. In almost every case, it is always working as a suppressor and a censor. We also seem to use it to pass judgement on whether the actions of others are "bad", but when we do so, what we are seeing is the effects of suppression on the thoughts of others taking "bad" actions. On the other hand, empathy is always inspiring us to take action. When we see others in need, we have a desire to help them and to perform a "good deed".

Components of the Moral Restraint Agency

The moral restraint agency censors and suppresses actions and thoughts which are "immoral". How is this agency structured? If we look at different people, we will see that they have vastly different "morals". For example:

Sarah, a four year old child, will do whatever she can get away with.

Richard, an adult, won't steal because it's illegal to do so.

Mary, an adult, won't steal because the Church says it's wrong.

Joe, an adult, doesn't steal because it doesn't seem right to hurt other people.

Robin, an adult, steals from the rich and gives to the poor
because the poor need the money more.

Robert, an adult and the member of a strange cult, doesn't see anything wrong
with stealing because he's been indoctrinated that everything is communal property.

All of these people are doing what they think is right, but they are doing different things for considerably different reasons. Sarah has learned that some actions have consequences; if she steals cookies, she'll get spanked, but only if her parents find out. Richard, Joe, and Mary don't steal, but for different reasons. Richard and Mary are both adhering to rules set by different higher authority. Joe is following what he believes to be right. Robert is adheres to the rules set by a higher authority, just like Richard and Mary. However, his higher authority has a different system with different rules. Robin, on the other hand, has placed herself above the morality of her society. She believes that she is justified in her actions because she is doing things for a higher cause.

Different cultures and communities have different moral standards. Different people incorporate these standards differently into their beliefs and their actions. For example, young children do whatever they can get away with, while adults often take their morals from a higher authority or build up their own moral system based on what seems right to them. Where peoples' morals come from and what the specific morals are can change independently. For example, if Mary believes whatever the Church tells her to believe, she would probably change her view on an issue if the Church slightly modified its doctrine (although the change might not be immediately apparent in her actions).

This gives us some insight into a possible structure for the moral restraint agency. At the bottom, we have many beliefs and desires. Some may come from higher authorities, some we may have made or reasoned ourselves, and some are simply things we are driven to do. Above this is a managerial agency which organizes our morals and beliefs and decides what to accept and what not to accept. The output of this managerial agency is then used to censor and suppress our thoughts and actions (see Figure 1).

Kohlberg's Stages of Morality

Different people organize their beliefs into morals in significantly different ways. The way our beliefs are organized (for example, whether we do whatever we can get away with or whether we take morals from a higher authority) has a significant effect on what we suppress and what we believe is right and wrong. Although the specific beliefs may vary greatly between individuals and between cultures, the way we organize the beliefs is much less variable.

A psychologist named Lawrence Kohlberg studied morality using many of the same techniques that Piaget used to study the development of common sense reasoning. Kohlberg presented subjects with a series of "moral dilemmas" and analyzed their reactions. It didn't matter which solution subjects picked, but what their reasons were for picking it. From what he observed, he developed a stage theory of morality with six stages (summarized from [1]):

Kohlberg discovered that many of the same results which applied to the development of common sense reasoning also applied to the development of morals. People seem to progress through the stages sequentially, with each stage relying on things learned in the previous stages. Higher stages can't be explicitly taught, but can only be learned over time. People also showed an inability to fully understand stages higher than their own. There were also some differences from the common sense stages. Different people progress through the stages at different rates, and most people never reach the highest stage. In fact, stages three and four are the most common among adults (see Figure 2). In many experiments, no subjects were identified to be in stage six.

The stages Kohlberg identified do not specify explicit morals or rules of conduct. Rather, they provide a managerial layer for organizing specific rules into a coherent whole. Without a clear organization, we would be wholly inconsistent in our moral judgements and would have trouble reaching conclusions.

Discrete stages are probably seen here for many of the same reasons that stages are seen in other areas. As we find that the organization of our morals is insufficient for dealing with the problems we encounter, we construct a new management layer (see Section 10.9 of The Society of Mind). When this layer is complete and can perform all the functions of the old layer, the new management layer replaces the old one.

During the early stages, we regularly encounter situations which our moral systems are unable to deal well with. This prompts us to move up to higher stages. When we start reaching higher stages such as four and five, there become fewer and fewer situations where our moral systems run into trouble. As a result, not everyone reaches the highest stages. In some ways this parallels the development of our theories of how things operate in the physical world. The common sense physics models most people have don't properly encompass orbital mechanics simply because we haven't had situations where our physics models fell short because of this lacking.

Is Morality Something We Are Born With?

Philosophers, religious figures, and politicians have been arguing for centuries over whether morals are something we are "born with". From our point of view, the question becomes somewhat clearer. A more relevant question is whether moral restraint is a separate agency in and of itself or whether it uses the same mechanisms used for other things such as learning and using common sense knowledge.

In the moral restraint agency, it seems likely that we learn a set of rules about what is right or wrong. Early on, the rules may be related to what causes us to be punished and what doesn't. Later on, they may be rules handed down from a higher authority. There's no real reason that these rules can't be learned using the same mechanisms that we learn other things with. There's no real distinction between "you must stop at red traffic lights" and "you shouldn't stick your hand into the toaster," although one is a "law" and the other isn't.

Above this layer of rules, the moral restraint agency has a managerial structure which decides what rules to follow and how to organize our decisions based on them. The structure of this managerial agency evolves in much the same way as the managerial structures for our common sense agencies. It is therefore reasonable to assume that they use the same mechanisms for their development. There seems to be a key difference, however, in that the output of the moral restraint agency is used almost entirely for suppression and censoring. One possibility is that although the moral restraint agency has evolved to be a separate agency, its learning mechanisms are actually governed by a separate outside agency (like a B-brain) which is also used for the learning of common sense knowledge.

It is critical that social animals develop some sort of morals. Communities would rapidly fall apart without any adherence to rules or suppression of actions. The development of social order, and hence some form of morality, is important to the survival of any social species. As a result, it would seem highly likely that some sort of mechanism would be evolved to ensure the development of a system that prevented individuals from always acting in their own personal best interests.

The Empathic Response Agency

The empathic response agency, the other side of the Society of Morality, seems to be totally separate from the moral restraint agency. Empathy itself, which is the response to the emotions of others, is seen even in very young children. The link between emotions and facial expressions, together with the hard-wired facial-expression recognition agencies, make it appear as if empathy itself is something we are born with. Empathy itself may be a very low level agency which is tied in closely with other similar low level agencies like anger, fear, and desires to sleep, eat, and reproduce. From our daily interactions with people, it seems like empathy has a strong affect on our own emotions. The anger of others often spills over onto us, as does their fear and their grief.

Empathy has many benefits to the survival of social animals. The ability for fear to rapidly spread allows herds of animals to quickly react to predators. The ability to sense anger allows us and animals to quickly judge the threat posed by an adversary. Most of these aren't related directly to the empathic response agency.

However, reacting to the needs of a child or the suffering of a comrade is beneficial to the survival of a society and the species to which that society belongs. It's this sort of thing which is a part of the empathic response agency. Through it, we react to the apparent needs of others and take actions to try to fulfill them.

Although empathy itself is almost definitely innate, the response to it may not be. For example, the empathic response agency may be learned as a way of reducing the desires generated by empathy. If it feels good to help people, we help people in a desire to feel good more often. Gaining the approval of others by helping people may be an added reinforcement. After this has been learned, we may help people even in cases where we get no reward from it.

The empathic response and moral restraint agencies do occasionally interact. For example, the desire to help someone might be suppressed by some moral restraint. On the other hand, a moral restraint might be lifted by an empathic response, particularly for someone in Kohlberg's second and third stages (e.g., someone might be willing to steal medicine in order to save the life of a friend.)


The Society of Morality gives us the tools we need to take actions which are not always in our own best interests. The moral restraint agency acts is reactive and suppresses and censors "immoral" actions or thoughts. The empathic response agency is proactive and encourages us to take actions to help others. As we grow and learn about things like common sense, our morals are also developing. The eventual results of our moral development leads to zealots, law-abiding citizens, patriots, children trying to please their parents, and even Hitlers and Gandhis.


[1] R. Brown & R. J. Herrnstein. "Moral Reasoning and Conduct." Psychology. Little, Brown, & Co., Boston, MA. 1975.

[2] H. Gleitman. Psychology. W. W. Norton & Company, New York, NY. 1991.

[3] M. Minsky. The Society of Mind. Simon & Schuster, New York, NY. 1986.

Erik Nygren (