[NOTE: I wrote this blog post a while back intending to put it up before the last several posts re: applying hierarchical cybernetics to political economics. Can't say why I forgot to post it, but perhaps it will help resolve some questions the readers might have developed in those prior posts. My apologies for any confusion.]
Actually it is a trick question. Nature doesn't ‘manage’ complexity. Rather, systems evolve to higher levels of organization that end up managing complexity by the emergence of structures at a new higher level that act to coordinate the activities of subsystems at the lower levels. For example, the emergence of multicellular organisms was a result of life evolving structural mechanisms that increased the cooperation between cells. As cells within these organisms became more differentiated (specialized in different functions) over evolutionary time, the organisms complexity increased leading to a need for some cells to first facilitate cooperative behavior and then evolve further to become specialists in acting as coordinators. For example the evolution of cells that are sensitive to photoperiodicity and their secretion of chemical messengers that trigger or turn off other cell activities is a good model of this phenomenon. The phenomenon is evolutionarily very old and can be found in both plants and animals. What is important to note is that as systems evolve toward higher complexity (discussed below) the need for better coordination between the parts grows. If the system is successful in evolving coordination specialists then the system, as a system, has an opportunity to survive in the long run. If not, internal instabilities and/or competitions will destroy its integrity and it will cease to function as a whole.
The question that should be posed is: How do systems of coordination evolve such that as complexity increases systems maintain their capacity to remain systems? Biological evolution is the paradigm example, but as we learn more of socio-biology we come to realize that human societies and their economic activities fit this model quite well.
What is Complexity in Natural Systems?
Complexity is a problematic word! There may be as many definitions of complexity as there are people who identify themselves as complexity theorists. So I will throw in my opinion on the matter. Then I want to explore the important issue of how complex systems are managed.
The definition of complexity that I want to advance is rather technical. I am making this explicit in the textbook Principles of Systems Science. Here I will provide a summary of what we propose there to give you a sense of how we can look at the problems associated with complexity in society.
Consider a system as a bounded set of component parts, down to some ‘atomic’ level. This may seem like an extreme reductionist program, but in fact it is a necessary starting point to grasp concepts such as emergence and evolution — much more holistic views. Atoms, however defined, come in many different kinds (for example in physics/chemistry elemental atoms). Each kind has common features, but at the same time some of these features have multiple variations (e.g. physical atoms all have electron shells but each kind has a different number of electrons and different numbers in their valence shells). Atoms of different kinds connect in ways that create combinational entities that are slightly more complex (e.g. molecules). These connections depend on energetics (e.g. covalent bonds in chemistry store energy).
While the fundamental example I used is actual elemental atoms and chemistry, the concept applies more universally to all levels of organization. Chemistry is important in the model of how life emerged from some kind of primordial soup. But then, at higher levels of organization, say prokaryote life, the cells themselves become the atoms and interact on their own level. Individual human beings are the atoms of social systems. Their interactions are enabled by energy flow and when chance encounters occur, provide opportunities for new combinations of social structure to emerge. So the use of the term 'atom' depends on the level of organization being examined.
To generalize, atoms (of a system) have multiple ‘personalities’ that means they will interact in multiple different ways. Complexity can now be defined as a set of atomic types, along with the total numbers of each type and a set of potential interactions (see Fig. 1). This is actually what I like to call ‘potential complexity’ as opposed to ‘realized complexity’. The former describes an amorphous system prior to the flow of energy (i.e. at thermal equilibrium). When energy is pumped into the system it starts to drive actual interactions as the system moves further from equilibrium. Structures begin to develop, some of which are energetically stable under the conditions of the energy flows (see dissipative systems), thus forming more complex objects that have their own potentials for interaction (see Fig. 3). Realized complexity is time dependent and must be specified in terms of actual interactions (e.g. actual bonds formed) at any given point in time. Systems undergoing stable energy flows over very long time scales evolve their realized complexity to some ultimate state, which then constitutes a steady-state condition far from equilibrium.
Figure 1. Systems are comprised of sets of atomic types that combine to form slightly more complex ‘molecules’. At equilibrium (no energy flow) systems are an amorphous aggregation of these atoms. The number of kinds of atoms (only three different kinds shown here), the number of kinds of possible interactions between them (when energy is available), and absolute number of atoms in each kind constitute the potential complexity of a system at equilibrium. Slightly further away from equilibrium, as energy flow drives combinations and recombinations, combinations come into existence based on the above characteristics — what is possible — and eventually stable forms, each at minimum energy, obtain. These tend to have additional interaction potentials based on geometry and ‘unused’ interaction potentials (see Fig. 2). Note that combinations of atoms introduce geometric considerations, i.e. the “shape” of the combination becomes an important factor in further development.
Figure 2. Stable associations of atoms, under the conditions of energy flow, can begin to form yet higher level associations. Pictured here are bonds between two geometrically compatible ‘molecules’. The red interactions are generally weaker (have less energy) than the interactions holding the atoms together in either object so that the objects retain their identity as subsystems. However, it is also possible that the between-object interactions can be stronger than at least some of the internal interactions which can cause reactions that produce new entities. For example, many chemical reactions between molecules are of this type. But the general principle applies at every level in the organizational hierarchy, even for societal institutional interactions.
The description of how new structures emerge, so far, suggests a rather linear course of complexity evolution. A graph of complexity vs time at a constant energy flow, theoretically, is a logistic, S-shaped, technically a linear system. But in the real world atomic and higher level interactions can include some non-linear aspects, such as an all-or-none reaction. Chaotic, critical, and catastrophic subsystems can dramatically change the general linearity of system evolution. Indeed, these kinds of non-linearities create opportunities for systems to make dramatic evolutionary jumps in organization that, as we will describe below, result in better managed complexity without suffering some of the problems associated with the increase in complexity as evolution proceeds. Figure 3 shows the comparison of smooth (logistic) evolution to a more chaotic (realistic) version. The curve in Fig. 3B shows some dramatic leaps upward and even some possible declines in complexity due to catastrophic effects. This version may be closer to what Stephen J. Gould described as ‘punctuated equilibrium’ for biological evolution. But just as the low level patterns of combination formations, as given above, are true at higher organization levels, so too is the form of the graph in Fig. 3B true for those higher levels. Even social organizations show similar fits and starts form as they tend to grow.
Figure 3. A smooth logistic (S-shaped) increase in complexity in an evolving system (A) versus a highly non-linear increase (B). The latter may even include shorter periods of decline in complexity due to catestrophic events. However, as long as energy flows and there is ‘excess’ available energy the trend will be upward until all available energies participate in the maintenance of system complexity after which a steady state level of complexity emerges.
Natural systems evolve to higher levels of organization by undergoing the odd occasional shuffle in internal organization at lower levels due to non-linear interactions, changes in energy flows, or introduction of new components from outside. Such occurrences lead to new structures and functions that then are subject to internal selection. Then, on perhaps even rarer occasions novel structures emerge from selection and interact with other components at that level of organization. As shown above, new interactions can lead to yet higher levels of organization. The whole system is more complex but also subject to some new possibilities.
Evolution is a drive toward higher organization and manifests itself in the form of organizational hierarchies. One way of looking at this is that material evolution, taking place in the thermonuclear fires of stars and the explosions of supernovae gave rise to chemical evolution in the radiant fields of stars (in dust clouds and on solid bodies like planets). That in turn, again in stellar radiant fields bathing planets (like Earth), gave rise to biological forms, which then began the Darwinian evolution we are most familiar with. That evolution produced ever more complex forms, eventually leading to primates and us. Then emerged the social organizations seen in some of our primate cousins but most complex of all in us. Our biological and social forms gave rise to complex technology and complex institutions. Roughly that is where we are today. All along the way the emergence of higher order complexity was based on the establishment of lower order complexity along with some kind of chaotic trigger leading to possible short term decrease in complexity but, so long as energy was available, the eventual long-term increase in complexity (see for example, Prigogine, Ilya; Stengers, Isabelle (1984). Order out of Chaos: Man's new dialogue with nature. Flamingo. ISBN 0006541151. Also, Morowitz, Harold (2004) The Emergence of Everything, Oxford University Press, ISBN13 978-0195173314.)
The element of chance plays a role in the evolution of systems, of course. It is relatively easy to see that systems with very large potential complexity measures have many different possible outcomes in terms of specific subsystems obtained at any given level of energy flow (and availability; energies that only contribute to thermal modes will be destructive rather than constructive). The number of different feasible combinations at a given level of energy flow at steady-state is a measure of the uncertainty associated with making predictions regarding the precise configuration of a system in evolution. On the other hand, some combinations are inherently more stable (having longer lifetimes) or more easily made so that not all combinations are equiprobable. This gives rise to the evolution of conforming patterns that may have many variations on common themes but universal (highly conserved) features nonetheless. Readers might find The Law of Requisite Variety of interest for more on this topic.
Complexity by itself is not necessarily a benefit to the system in the long-run. The problem with complexity is that it is a reflection of diverse specializations of components. And that can be a problem when there are a large number of components, each trying to do its own thing without paying attention to what other components are doing.
Enter the need for coordinators. As with the example of photoperiodicity and the role of special photosensitive cells in regulating the activities or timing of activities of other cell types, components at any level of organization may evolve as regulation specialists. The animal nervous system is an obvious example. Animals with brains, even primitive ones, show how special tissues composed of neural cells evolved into specialist information processors, taking in messages from sensors and molecular moieties in the blood, computing a response, and sending command signals to other tissues to cause them to behave.
All tissues/organs have local control through various kinds of closed-loop controllers, feedback regulation that keeps them performing within specifications. But whole bodies consist of many different kinds of specialized tissues and organs that must work in a coordinated fashion if the animal is to be healthy. The first evolutionary stage leading to body level coordination was the development of larger-scale homeostatic mechanisms where one tissue (generally located at the anterior, or head, end of the animal) monitors and signals another tissue when the latter needs to operate in a different regime from normal. For example the nervous system may process stressful conditions and signal the adrenal-producing tissues (e.g. the adrenal glands) to release adrenaline in order to boost the activities of other tissues (e.g. heart and breathing muscles) to respond to the stressor and possibly reduce the stress level.
A sufficiently complex system needs a considerable amount of internal coordination. In the hierarchical cybernetic model this is called logistical coordination. It depends on specialized coordinators that monitor multiple subsystems and issue commands to achieve an optimal level of activity for the whole system even when this means sub-optimal performance for one subsystem. But also sufficiently complex systems are such because they are interacting with complex environments. And the more complex an environment is, the more stochastic (uncertainty) it is, leading to difficulties in those interactions. Systems need to obtain resources from the environment and expel wastes into that environment. That means the system must coordinate its own behavior with entities in the environment.
Living systems that have to move about to make a living (animals) evolved elaborate external sensory apparatuses to obtain information about what the environment was doing. They evolved more elaborate brains for monitoring this flow of data and computing the information content. And those new coordinator processors, called “tactical coordinators” issued commands to the muscular system (actuators) to drive external behavior.
In very large systems multiple such cooperation facilitators emerge, all with different spatio-temporal scopes and as often as not, competing among themselves for resources and lower-level objects to coordinate! Given enough time and energy these higher-level objects will find coordination options and yet again a higher level super coordinator(s) will emerge to coordinate the coordinators. The evolution of all living systems shows this pattern.
For a more in-depth look at this recurrent emergence of coordination, please see my working paper on Cybernetics and hierarchical control.
So-called super-biological systems (i.e. societies and ecosystems) can sometimes show a similar pattern. Primitive human societies clearly evolved a hierarchical management structure that became more elaborate (complex) after the advent of agriculture when coordination specialists formed layers of “officials” and militaries formed to protect the agricultural lands and populations. In the case of ecosystems that have reached or nearly reached a climax state and have been stable for long periods of time, the establishment of stable food webs with clear trophic levels have achieved a form of internal coordination. My favorite professor at the University of Washington, when I was an undergraduate in zoology, Robert Paine developed the notion of a “keystone species”, being a species that has a remarkably large impact on the balance and functioning of a specific ecosystem. I see this concept as the emergence of a regulatory feedback mechanism that acts as a coordinator over a significant portion of other species in the system. Ecosystems, however, do not have, as far as I can discern, any kind of tactical coordination functions. They do not have a way to sense the other ecosystems around them. Where they exist as clearly delineated systems, they are bounded by geographical and climatological conditions. They do not directly interact with other ecosystems except for the migrations in (invasions) and out that change the mix. Thus ecosystems are subject to possible disruptions from other ecosystems (especially, for example from the human ‘economic system’).
I maintain, however, that human social systems have not really evolved (culturally) true hierarchical cybernetic systems to coordinate the society's economic activities. What we call government is only a poor shadow of a functional hierarchical cybernetic system. And as I argued above the latter is absolutely essential to manage complexity and have a long-lived whole system — in this case our global civilization.
Societies have continued to evolve complexity through the invention of technologies that have accelerated the process. Today our cultural complexity far outstrips our ability to manage our society. This is the fate of coevolution between two systems with wildly different time constants, in this case between the human genome and human culture.
The Problems for Societies Associated with Complexity Increase
The reason this is important is that we are beginning to recognize a diminishing returns phenomenon relative to the increasing complexity of society. Joseph Tainter wrote an extraordinary book called “The Collapse of Complex Societies” (1990, Cambridge: Cambridge University Press. ISBN 0-521-38673-X.) in which he details how societies throughout history have evolved through the process of ‘problem solving’, which process led to increased complexity of institutions over time. He analyzed the way in which that increase in complexity eventually had lower payoffs, either by creating new problems (which then needed to be solved by increasing complexity!) or simply the fact that new solutions could not completely produce the needed results. Eventually, he pointed out, the benefits of increasing complexity peaks when the costs exceed the benefits and begin to dominate the process.
Another factor affecting complexity of society is the invention of technologies that then require more care and nurturing (leading to new institutional requirements for effort to be expended). We generally think about technological developments making our lives easier, saving time and energy. But the fact is that each development carries many hidden costs in these areas. Technology users often find themselves taking more time attending to the requirements of operation (think how the Internet improves access to information, but how much time gets spent searching for that information). But more importantly, each advance in technology requires an army of technical support people and organizations to provide the services. Each advance adds to the general knowledge base and that base requires maintenance. In general, advances in technology contribute to increasing complexity.
According to Tainter, then, at some point increasing complexity starts costing more than it contributes. Unfortunately, the human response to the problems arising from increasing costs is to try to solve those problems by increasing complexity even more. Thus this leads to a positive feedback that accelerates the decline and possible collapses of societies. Decline (and eventual collapse) is an inevitable result of increasing complexity in that society. History bears out that societies have developed and declined, so the evidence is rather strong that this theory has a lot of merit.
Thomas Homer-Dixon has also advanced a highly related factor in social collapses having to do with the the flow of energies through the society. Complexity, is both driven by energy flow and enabled by it. That is, the availability of excess usable energy tends to find ways to be used! Humans are motivated to trade their own efforts for machines that take advantage of this excess (extrasomatic) energy and so are motivated to find inventions that allow doing so. Excess energy always tends to be used to increase complexity! This phenomenon is actually not unique to humans and technological invention. As I described above, the evolution of system organization is driven by energy flow and that organization that is stable (stabilized by evolutionary selection) contributes to energy being directed toward more useful work. Here useful work means building greater complexity. Now, as Homer-Dixon points out, any reduction in energy flow, such as when the Roman Empire experienced deterioration of its food and timber imports from the distant conquered lands, results in an inability to either generate new or maintain old complexity.
Homer-Dixon and Tainter are both addressing different views of the same phenomenon. Energy flow drives the tendancy toward increased complexity within the system up to the point where there is no more excess energy in the flow. Afterwards, in the best case where energy flow is maintained in a steady fashion, the system could, in theory, maintain a steady-state level of complexity as in Fig. 3A above. But what happens if energy flow starts to decline as Homer-Dixon describes for the Romans? Diminishment of energy flow diminishes the capacity to maintain that level of complexity, which leads to diminishing support for the system and eventual collapse. Without continuing sufficient flows of energy, the attempt to increase complexity (solve problems) that Tainter talks about can only result in declining results. And that is the way in which collapse manifests itself.
Today, for global society this paints a grim picture since we are in the early stages of the decline in major energy flows from fossil fuels. Our society, our technologies, our population size, and our ability to have machines substitute for our personal physical efforts have been made possible by the flow of excess available energy from those fossil fuels. Nothing else comes even close to providing the power needed to maintain our society.
Diminishing returns on complexity and the threat of declining energy form dual problems for the continuation of any system. At the same time we have ample examples in nature where systems have reached a level of complexity and declines in energy flow that should have led to collapse, and yet they did not. Rather the system managed to completely reorganize in such a way that the negative effects of peak complexity were mitigated, allowing the whole system to continue operating in spite of, for example, lower energy flows. One can argue that what continues is really a different system, but the basic functionality of what that system performed do continue. Biological evolution provides many examples of this process of reorganization. What interests me is how can we learn from nature, how complexity can be managed such that it doesn't result in collapse per se.
The Evolution of True Coordination in Human Society
Can biological evolution catch up with cultural evolution? What needs to happen to achieve true hierarchical cybernetic management of the human society, the political economy?
If I am right about the fate of human evolution being adversely affected by the advent of agriculture, and if our technologies cannot prove to protect us from extinction, then an explanation for why we are not operating in a true coordination system is that we are simply not yet evolved to true sapience. Why? Coordination and the whole hierarchical cybernetic framework is based on communication and information/knowledge. The human modes of communication are a mix of inherited body and facial language (emotional) and symbolic, recursively productive language. These are inherently noisy and with our much less than sapient brains subject to a lot of misinterpretation. Couple this with the fact that low sapience means that our emotional and especially our self-serving side is quite predominant in our dealings with one another and you have the basis of a very imperfect information system. For a cybernetic system to work you must have reliable information systems.
I can think of no better example of an unreliable information system coupled with a selfish psyche than the current state of the monetary system. Money, in its purer form is really just an information system that conveys messages about how much available energy there is for work (or also how much work has already been accomplished). Today money has lost its role in this regard because non-sapient, selfish humans have figured out how to temporarily debase its information quality so as to make it seem they have more of it and therefore are wealthy. And of course everyone else falls for it so they get by with it. Our emotional and cognitive relationships with money are a mess. It has become the end-all and be-all of our worth.
Frankly the only way I see for biological evolution to catch cultural evolution is if either we halt cultural evolution, say through the collapse of society, or speed up biological evolution by intervening with, maybe, genetic engineering. The problem with the latter, of course, is that we have no idea what we are doing. We're not sapient enough to know if that would wise. We've certainly raced ahead with many other technological solutions that have come back to bite us in the collective arse. More likely, in my opinion, our civilization will collapse under the weight of population, overbearing complexity, and diminishing energy flows. If there are survivors, maybe they can give humanity a chance to finish the business of evolution naturally. My speculation is that if humans were ever to evolve greater sapience (and I'm convinced that the evidence for this possibility is quite strong) then a natural hierarchical cybernetic system would emerge in their socio-economic systems. They would be able to communicate (for example would have much greater empathy yet much better control over emotions) and would see to it that their communications provided real information. Our current species is a lot closer than you might think when looking around at all the foolishness found in our world. With just a little more time and the right environment...
 The whole subject of energy flow is quite technical. Among several issues there is the ‘kind’ of energy (e.g. photons, electron flow, etc.), the gradient potential (e.g. brightness, voltage) and the coupling with components, or how the energy affects the interactions between components (e.g. endothermic or exothermic). These issues are important and I do not want to ignore them, but they could fill volumes to do justice. The term “exergy” is a term that encapsulates all of these issues. Exergy is the energy available to do work. For example to form chemical bonds or move molecules (or people) from place to place along the gradient.
 Speculatively, sapient human beings could, I imagine, become the tactical facilitators for ecosystems. With enough knowledge of ecology and life in general, humans could monitor all of the potentially interacting systems and mediate in such a way as to help each system obtain its needed resources and disperse its wastes without harming the others. Which, if taken for the whole globe would amount to becoming the logistical coordinator for the Earth. Of course our species isn't sufficiently sapient and if we were we might discover that the Earth doesn't need a coordinator! What makes me speculate on this is that the emergence of coordinators has been the rule in evolution, not the exception. Ergo as the Earth perhaps approaches a steady-state dynamic, humans as the “brains” of the planet might not be so out of line.
 Of course any massive reorganization, while preserving the functionality of the system, is going to feel like collapse to any sentient participants. The end of the world as we know it will be traumatic. But it is important to understand that it is ONLY the end of the world as we have known it and not the actual end of the world period.