discordia concours

In the late 60′s, Marvin Minsky was trying to design a machine that could prove geometric theorems. He designed the algorithm and then sat down to simulate it using pencil and paper — and, to his own surprise, spat out a more elegant proof of the pons asinorum than Euclid ever knew. Now: did Minsky come up with the proof? No: he was simply functioning as the hardware here. Minsky came up with the algorithm, and the algorithm came up with the proof.

I like this story because it tweaks our intuitive notions of agency, and focuses us on the question of what processes produce what results. As Leibniz appreciated, mathematical innovation itself owes much to simply inventing more elegant procedures — new algorithms or heuristics which tend to lead to interesting results, reducing the need for ingenuity. Newton came up with the infinitesimal calculus and the calculus came up with his physics.

Aristocracy is just the allocation of power to the virtuous. “Meritocracy” has always been a redundant product of semantic drift, an attempt to evade the reality that aristocracy contains the seeds of its own destruction — that is, the Peter principle, which states loosely that “individuals within a hierarchy tend to get promoted to their level of incompetence.” This amounts to saying that meritocracy is impossible in the long run.

Pluchino et al (2010) runs some computational models showing that all you need for this to happen is aristocracy plus heterogeneity of tasks between levels. Which is hardly surprising, but it gets better: after getting a quantitative picture of how much inefficiency this causes, he shows that you can get better results with stochastic promotion — kicking workers upstairs at random. There are several ways to do this, but my favorite method is the one where you take the best performer and the worst performer and flip a coin to see who gets to be boss.

The idea behind this is simple to the point of bordering on tautology, and doesn’t actually require any assumptions about hierarchical structure: if there’s a high correlation between competence in one role and competence in another, then moving the best performers to new roles is unlikely to diminish systemic efficacy much. If, on the other hand, there’s a zero or negative correlation in competency across roles, then switching high performers out of their current positions is folly, since they’ll tend to regress to mediocrity or outright incompetence if you swap them around.

This actually means that the lower the mutual information between roles, the more we should tend to promote the worst performers, rather than the best. Whence the complement to the Peter principle, the Dilbert principle: that promotions to management are a mechanism for getting incompetent employees out of actual technical roles. We can see that they aren’t opposed alternatives — which one holds depends entirely on the correlation structure of skills and roles. (Note, though, that a high g factor weighs in favor of Peter.)

Start, as the ancients did, with an utterly unconstrained tohu va vohu — where anything can happen, and consequently everything does. Then impose a constraint: some events are nolonger allowed to happen. How do you measure how much any particular constraint reduces the variety of your world?

Wiener & Shannon worked this one out in the 1940′s, inaugurating what became known as the theory of information and control. (These are just two terms for the same thing seen from two different ends of a communication: information to the receiver is control from the sender.) Let’s say that you’ve got some x varying continuously within a given range, and a probability density p(x) over said range (i.e. a function defining what portion of its time x spends in any given state). Following Wiener, we can make a few innocent (and inessential) assumptions about the distribution and define a measure of the variety on x:

This gives us a nice quantitative sense of how much wiggle room x has within the given constraints — in other words, how uncertain we are about its value at any given moment. But what constrains a thing? The obvious answer would be “some other thing”. So let’s say we have a y constrained comparably to x, though it can have a very different p(y) and doesn’t have to be measured in the same units. Then we can form a joint probability density function for them and use that to get their joint variety:

You can think of this as measuring the total uncertainty on the couplet (x,y). These two things have some sort of relationship whose specifics we don’t necessarily know, but we can immediately measure the strength of the relationship thusly:

And this right here, folks is important: I(x;y) is called the mutual information of x and y, and tells you how tightly correlated any two variables are. A couple of things you may want to notice in passing is that it’s a symmetrical measure, so that I(x;y) = I(y;x), and that I(x;x) = H(x), so you can think of the variety on x as its self-information.

Now, Shannon was particularly interested in the special case where x and y were the inputs and outputs of a transducer — that is, a communication channel of doubtful fidelity. In this case you can think of the maximum value of I(x;y) as the capacity of the channel. One of Shannnon’s most important theorems (sometimes called the Coding Theorem) states that you can correct an arbitrary amount of error introduced to your signal by increasing the capacity of the channel (or, more usually, that given a fixed channel capacity there’s a limiting rate of transmission you can achieve without going over an error threshold). It doesn’t tell you how to do it, but it’s nice knowing that you can.

Ross Ashby was, so far as I know, the first person to see the potentially huge implications of this theorem, and generalized it into what he called the Law of Requisite Variety, which says that just as it takes money to make money, it takes variety to destroy variety.

Recall what I said above about control being just information seen from the outbound side, and think about the fact that being able to control your environment is a necessary subgoal to any other goal you might have — you need to be able to cancel out the noise your environment throws at you in order to accomplish anything. Now, I’m this animal with a bunch of inputs (senses and such) and outputs (muscles and whatnot), and my nervous system is effectively one big transfer function that coordinates them. As a necessary condition for me to live my happy little life, I need to be able to correct any disturbances my environment throws at me. Increasing my ability to adapt is isomorphic with increasing my channel capacity.

Looking at the three terms on the right hand side of the equation above, it suggests three ways that I can do that: increase the variety of states of the world that I can discern (perceptual range), increase the variety of ways I can respond to them (behavioral range), and decrease the joint variety between these two things by making my behavior more determinate — since it’s not so good if half the time I see a morsel of food I randomly shove it into my own eye rather than my mouth.

We’d expect animals that have made a living this way for a long time to have bred-in biases toward increasing their adaptive capacity in precisely these three ways — whence curiosity, play, and habit, respectively. But each of these impulses has failure modes — such as taking in a lot of useless information, wasting effort on pointless shit, and behavioral rigidity, respectively. And these should tend to fail more often the more our environment consists of other animals trying to do the same stuff. What works well against a relatively passive environment fails pathetically against one that’s trying to control you as much as you’re trying to control it.

The consequences of this will be elaborated subsequently.

[This post will likely undergo repeated revisions for clarity.]

“For a student to learn Newtonian physics is a creative act comparable to Newton’s original invention. The main difference is that the student has stronger hints than Newton did. The conceptual transition from the student’s naive physics to the Newtonian system recapitulates one of the great scientific revolutions, rewriting the codebook of the student’s experience. This perspective has greatly increased my respect for the creative powers of individual students. It is antidote for the elitist view that creativity is the special gift of a few geniuses.”
– David Hestenes, “Reforming the Mathematical Language of Physics” (2002)

If Quigley and Tainter are Pestilence and Famine, then Peter Turchin and Mancur Olson are War and Death, respectively. Olson’s The Rise and Decline of Nations will tell you how self-serving economic factions kill the body politic much as cancer kills the body organic, and how a moderate catastrophe can actually benefit long-run economic growth by breaking up rigid patterns of entrenched interests. What he teaches is that cooperation’s virtue is in the eye of the beholder, as he demonstrates how small, cohesive groups cooperate to exploit larger, less cohesive groups.

He reminds me of Uncle Fritz in La Gaia Scienza (s. 295): “Yes, at the very bottom of my soul I feel grateful to all my misery and bouts of sickness and everything about me that is imperfect, because this sort of thing leaves me with a hundred backdoors through which I can escape from enduring habits.” (If you want to see the intrapersonal analog, check out George Ainslie’s Breakdown of Will, particularly the last few chapters.)

But Turchin, with oodles of data and ecological modeling techniques backing him, homes in like some kind of nerdy Sherlock Holmes on the key variable that seems to make or break a civilization — and finds that Ibn Khaldun was right. Solidarity is the glue that holds a civilization together as it expands, and when its supply is insufficient or poorly distributed the entire social structure becomes ataxic.

You can think of asabiya as a measurement of a system’s capacity for coherent action. It implies not merely that the system holds together but will hold together robustly in the face of disturbances; not merely that its parts are coordinating but that they will persist in doing so under a wide variety of circumstances. This may sound like a variable too squidgey to be usefully quantified, but this is not so: it can, in fact, be rigorously defined and empirically measured — you can even do calculus on it. But this is for later.

Taken together, the Four Horsemen define the problem a hypothetical guardian angel has to stare down and grapple with. The first bitter pill to swallow is that in order to avert major crises one may need to foment small ones; the second is that cooperation is a feature of the world that’s as uncaring and morally neutral as the weak nuclear force.

Previously I’d mentioned Carroll Quigley‘s analysis of institutions themselves as engines of civilizational collapse. This actually meshes well with Joseph Tainter‘s analysis in The Collapse of Complex Societies, wherein he posits a simple economic model: increasing production and population requires institutional apparatus to manage; the increasing overhead costs of institutional superstructure create diminishing marginal returns to increases of production, and the closer to equilibrium the social system is the less able it is to respond to shocks (e.g. sudden energy shortfalls).

What’s great about Tainter’s thesis is that rather than trying to explain specific collapses via a bunch of disparate shocks, it shifts the focus to the question of why those civilizations were vulnerable to shocks in the first place. The exogenous, proximate causes are things we can’t control much anyway, so better to focus on the endogenous, ultimate causes. What sucks about this point of view is that it paints a pretty dire picture where civilizations end up in a giant Chinese finger-trap, where “downsizing” simply cannot happen without a catastrophe.

But the inevitability of it all hinges entirely on the dependency of actors on the institutions that are creating the instability. If you can find a way to seamlessly shift people away from such dependency, such that when the next big shock happens they can find ways to get their needs met without all the bureaucratic overhead — well, the institutional structure will still collapse, but there’ll be another system in place to migrate to. (Just make sure it scales well!) Which is a pretty okay picture of graceful failure.

Something unprecedented happens, and people have to decide how to respond to it. There are two poles between which attitudes will subsequently tend to be distributed: “that was an exceptional event, and now we can forget about it and go back to business as usual” versus “let’s codify a policy to deal with any and all future instances of anything that looks remotely like this”.

Both responses are motivated by the trauma of being faced with the unpredictable, and neither is obviously correct. Both are magical thinking insofar as they assume that you can control the unseen by, respectively, cognitively boxing it out (“taboo”) or boxing it in (“ritual”). The kind of neurotic denial that can arise from the former is obvious, but the pathologies of the latter are less well recognized.

This is, for instance, where bureaucracy comes from; bureaucracy rarely springs forth fully formed from the Abyss, but rather accretes insidiously by a series of over-generalizations and hyper-corrections (“policies”) that build one upon another, until you need a whole support staff to maintain the creaking kludge of a structure.

Carroll Quigley, to my knowledge, is the only historian to put his finger firmly on this as a pathology of civilization: rules that begin as instruments for collective decisions end as institutions that serve primarily to further their own existence, at which point the civilization starts to cannibalize itself.* What nobody, to my knowledge, seems to have pointed out clearly and forcefully is that this higher-level drama corresponds directly with a lower-order one going on inside individual minds.

But there is a middle way. Common law, for example, gets around this by accumulating cases which are then used to inform judgments on subsequent cases, with judges continuously tinkering away on conventional wisdom. The notion that law is about universal rules is an encroachment of bad philosophy (think Plato and Kant) into a practical art, and is actually antithetical to justice (in the classical sense) insofar as it attempts to replace judgment with automation. An armory of well-tested heuristics, a well-ordered memory for facts, and a tolerance for ambiguity make for more adaptive decisions than trying to decide everything by algorithm.

What seems to make the difference here is whether you tend to think of decisions as permanent or provisional. Buddhism nails this one nicely at the fundamental level: if you grok that everything is open, you can respond with agility to what’s actually happening and not worry too much about trying to predict everything. What you have to sacrifice for this power is an addiction to perfection.

* Anyone who doubts that we’re already at this point should reflect on the fact that nearly everyone working in any modern governmental apparatus is unelected and effectively unfireable.

The tl;dr of this paper is that an expectation of being interrupted makes you more akratic: all else equal, you’ll take smaller, nearer rewards over larger, further ones when you have a higher expectation of being interrupted in your search for rewards. We leave it as an exercise for the reader to cash out the implications re: the effect of cell phones (&c.) on one’s propensity to reap the rewards of deep concentration, etc.

“Entrepreneurship” is an idiotic category that gives people all the wrong connotations about what it takes to make a living doing something original: you don’t need to be a Type A trailblazer with a grand vision and a strong preference for risk. All you need is a clear sense of what you’re exceptional at (or could be), what motivates you powerfully (or could do), and a way of transducing these skills and motives into tangible value — a craft. These aren’t trivial requirements, but they’re also totally tractable, and are made to seem harder to meet than they actually are by the fact that kids aren’t given instruction on how to find their craft.

Because, you know, that would lead to, like, anarchy or something. I mean, can you imagine a world where everyone is gainfully self-employed at something they actually give a damn about for its own sake? How on earth could we justify bureaucracy then? The org charts would look like spaghetti code! Horreur! But, we do need at least some creative activity going on to enable all this economic waste heat. Better to just place the people who’re genetically predisposed to this kind of behavior on a pedestal, treat them like mutants and suggest that there’s some magic fairy dust or anointment that marks them apart.

The joke about French having no word for “entrepreneur” is actually darkly funny — the conspicuousness of the word in our own vocabulary is itself a sign of pathological thinking. An artisinal society where entrepreneurship was as common as dirt wouldn’t need it.

The ideal computer is reconfigurable in every conceivable way, and never lets the operator forget this fact. For instance, it shouldn’t merely permit the operator to fully customize every aspect of its interface, but subtly encourage the operator to refactor the interface on the fly, tailoring it to optimize flow in accordance with the op’s changing needs and cognitive style. The fact that most users don’t even know about keyboard shortcuts is an indicator of how far short your typical computer falls of this ideal.