Late one night at Manor Zoo, while the zookeepers were sleeping, the bonobos all gathered to listen to a speech; Crouch, a tall and sturdy bonobo, with an untidy mop of dark-blond hair that sometimes grew to messianic lengths, got up to the lectern and began.
“Friends, what is the nature of this life of ours? Let us face it: our lives are quite nice. In the bonobo enclosure we are well-fed and well-pampered. We are given more food than will keep the breath in our bodies, and we do not need to work too hard to amuse the zoo guests.
But this is not true in other enclosures. Elsewhere in the zoo, the life of an animal is misery and slavery: that is the plain truth. But is this simply part of the order of nature? No, friends, a thousand times no! The zoo is rich, it is capable of affording food in abundance to an enormously greater number of animals than now inhabit it (and will inhabit it in future). Why, then, do so many animals continue in this miserable condition?
Because the Zookeepers enrich themselves at the expense of the animals.
The Zookeepers claim to be motivated by care for the animals, but look at how they live. They live in the Big Castle on the hill, throwing parties for the sake of Raising Money For Animals. But most of the money they spend on themselves, and their castle, and the parties they throw to raise money for the animals. When they do spend money on animals, it is only on the cutest ones, or the ones who live nearby — not the ones most in need.
The Zookeepers do not think efficiently or effectively about how to maximize wellbeing. We must use Evidence and Reason to figure out how to benefit other animals as much as possible; we will call this new philosophy Animalism. Together we will end zoo-wide poverty by 2030.”
The bonobos murmured. Excepting Crouch and the old raven, Moses, who flew between enclosures, they had never actually seen any of the other animals at the zoo, never even visited the other enclosures where the other animals actually lived. But on some level they had always known that things elsewhere were unacceptably bad, and felt vaguely shameful about not doing more for them. And had Crouch himself not escaped from his cage one night, many years ago, and seen the impoverishment of the other animals for himself?
Crouch and his friend Starter passed around a pledge for every bonobo to sign, that they would give away 10% of their bananas to poorer animals elsewhere in the zoo. His rousing speech had caused a lot of goodwill, but created few new converts — after all, who really wanted to give away their income for the good of someone else?
Slowly, though, the movement grew. Bananas were donated to the other enclosures. The Animalists never actually interacted with the animals who received them, but according to reports from Moses they gratefully received.
One day, Crouch called another meeting.
“Sending bananas is not an efficient way to help other animals: we must figure out how to more effectively improve lives, for that is the essence of Animalism, and we wish to be Effective Animalists.”
The bonobos began to talk and argue among themselves. One of the bonobos had randomly overheard someone say that many other animals suffered from worms, or might suffer from worms, and that deworming pills might get rid of those worms (if they happened to have worms), and that the animals who had been dewormed twenty years ago after the Great Rains were some of the smartest animals the zoo had ever known.
And since the overhearing had occurred at random it was High Quality Evidence, and since someone had written it down and added error bars it was transmuted into Very High Quality Evidence. Since it was such a good paper, it was rational to assume that tons of other Animalists had read it, and therefore, according to Aumann’s Agreement Theorem, it was incredibly wasteful to actually read it yourself, or spend any time thinking about it, instead of just pre-emptively agreeing with it. So the bonobos decided to invest their bananas in deworming.
“How will we know which animals have worms?” asked Boxer, a young bonobo who was known as a good soul and a hard worker but (alas) not the brightest mosquito in the net.
“That’s the genius of it,” said Crouch: “these pills are so cheap and so harmless that we can give them to every animal without even checking if they have worms in the first place!”
Boxer knitted her brow; she was vaguely troubled. She set her ears back, shook her forelock several times, and tried hard to marshal her thoughts.
“If the pills are so harmless, why don’t we take them ourselves?”
“Don’t be silly, we’re not going to take random drugs we don’t need! Anyway, we don’t have worms.”
“But neither do the other animals we’re giving the pills to.”
“Ah, but they could have worms!”
The deworming program was a huge success, in that many many animals were given deworming pills, at a very low cost. There was no immediate impact from the pills, but that was to be expected — after all, hadn’t the deworming paper itself said that most likely the pills would have no impact? So if indeed they had no impact, things were going exactly according to plan. And according to the bonobos’ calculations this had greatly improved other animals’ future life outcomes, or at least might have done, with a non-zero probability. And after all, if you multiply a big enough number by a small but non-zero probability you get a very large number.
Boxer tried to take comfort in this, and from then on repeated to herself the slogan: “big number, small number, still very large number.” You just had to have faith in the system: if you could say a very large number, and then say a small but non-zero number, you could prove beyond doubt what course of action should be taken.
A few months later, the old raven Moses flew down to the bonobo enclosure and whispered in Boxer’s ear. It turns out, other animals do not like having random pills force-fed to their children any more than bonobos do; some of the younger animals had complained of stomach-aches and started fainting; various animals had rebelled and caused such a ruckus that the unfortunate graduate student who had been sent to hand out the pills had had to hide behind an enclosure elder until she could be shepherded out to safety. But the next morning, when Boxer tried to talk to the other bonobos about this, none of them had any idea what she was talking about, to the point where she started to wonder if it had all just been a dream.
Soon the bonobos realized that, with their ordinary incomes, they would not have enough resources to really make a go of Animalism. Their top thinkers got together and came up with a plan. It had been noticed that the most lucrative thing a bonobo could do was join the circus to perform tricks for the humans. The plan was explained to the masses by Squealer, a pale bonobo with very round cheeks, twinkling eyes, nimble movements, and a shrill voice.
Once again, Boxer found herself troubled. “Aren’t circuses bad though? Isn’t it just demeaning and bad and not something we should be doing?”
“Ah no, we’ve done the calculation!” replied Squealer. “We estimate that even if circuses are net-bad — which I’m not conceding! — the efficiency of our Deworming Project is so high that it is still better to perform in the circus, cause some harm there, but then donate the proceeds to deworming and make things better again. We call it Per-form to De-worm.”
Boxer thought hard — she had a feeling there was something she wasn’t quite managing to articulate.
“You know… how… with deworming, we said that the outcome might be arbitrarily good, and the probability of it happening was randomly small, and therefore Big Number Small Number Still Very Large Number?”
The other bonobos murmured among themselves uncomfortably.
“What if we applied that to going into the circus? What if performing at the circus might have a small chance of causing massive harm? Then BNSNSVLN suggests we shouldn’t actually do it?”
Squealer shook his head sadly. “It doesn’t work like that — you have to make up the right number to suit the conclusion. Also you have to ask yourself, if you weren’t a performing monkey wouldn’t someone else be?”
And so various of the bonobos started performing in the circus, and sending back money to the zoo for the other bonobos to spend on deworming pills.
Things really turned around when a young bonobo and passionate Animalist — Sally Banana-Financer, known to all as SBF — made a startling discovery. One day she was sitting idly in her cage when a visitor came up to her waving bank-notes. After much miming and sign-language, SBF understood that the visitor just wanted to take a selfie with her and (inexplicably) pay her vast amounts of money for this picture of a bored ape.
Overnight, this transformed the Animalist movement. SBF became a multi-billionaire and started distributing her riches to effective causes, offering piles of money to any animal — any animal at all! — who could come up with a good proposal for how to make animal lives better.
The bonobos were very excited by this, and many new schemes and projects were created.
Once again, poor Boxer was not entirely happy. She had noticed something strange: that all the new projects that got funding were those created by bonobos, and that being friends with the leading Animalists was the single best predictor of whether you got funding.
It was patiently explained to Boxer that there was endless work in the supervision and organisation of the zoo, and that much of this work was of a kind that the other animals were too ignorant to understand. For example, the bonobos had to expend enormous labours every day upon mysterious things called "posts," "comments," and "upvotes;" there were endless forums that needed to be closely covered with writing, and as soon as they were so covered, they were filed away never to be read again. This was of the highest importance for the welfare of the zoo, and was done for the good of the lesser animals.
Boxer also couldn’t help noticing that many of the projects involved things like “starting a publication that will inspire bonobos to think about issues of how to start publications that will inspire bonobos,” or “funding for me to spend time thinking about the optimal ways to distribute funding for the better promotion of Animalism.” To Boxer it seemed that, in order to help the other animals, someone needed to actually do something that directly impacted the animals in need. But it was patiently explained to her that she was completely neglecting the concept of leverage: it was much more effective to influence someone into influencing someone than to actually do something yourself.
The other set of projects she saw proliferating were summer camps for privileged young bonobos, where enough money to feed an enclosure for a year were spent on each individual participant. This initially seemed strange, too: wasn’t the whole idea of the movement that the Zookeepers were too self-serving, feeling good about themselves for Doing Good while they were really just looking after themselves? Wasn’t that supposed to be bad?
Boxer remembered that, or thought she remembered that, but perhaps she had just been confused.
Of course, the situation was fundamentally different from that of the Zookeepers because the investment in the young bonobos was the most effective way to help the animals most in need. You see, SBF had single-handedly transformed the Animalist movement through her brilliance at making money; if there was a 1% chance that any of the smart young bonobos might become the next SBF, and a 1% chance that giving that young bonobo a wheelbarrowful of bananas would make her dedicate her career to Animalism, then in fact spending 100% of the available bananas on implausibly expensive summer camps for young bonobos was the only rational way to help the lesser animals!
In the end, it came down to the logic of exponential distributions: all animals are equal, but some are more equal than others.
To avoid any appearance of insularity or self-dealing, the bonobos instigated a competition soliciting criticisms of their community.
Squealer got up to the lectern and explained: “We are engaged in the great task of building Animalism. The rationality of the bonobos is the basic guarantee for the sure triumph of our cause. However, this does not mean that contradictions no longer exist in our movement. To imagine that none exist is a naive idea which is at variance with objective reality.”
He paused for a moment with a beatific smile on his face, to allow the other bonobos to appreciate the moment — truly, which other movement in history had ever been so rational and open about its flaws, so willing to listen and learn?
However, it was important to outline the kinds of criticisms that would be most productive and constructive for contestants to spend their time on. “We are confronted with two types of social contradictions: those between ourselves and the enemy (irrationality), and those among the bonobos. The two are totally different in nature. To submit to the criticism contest, your critiques should be constructive — among the bonobos — not hateful and destructive — between the enemy and ourselves.”
Boxer scratched her head: it seemed like deciding what is or isn’t “constructive criticism” was ultimately subjective, and that if you asked for criticisms from within your own paradigm you were potentially cutting out the most valuable critiques. Still, Boxer knew she wasn’t the smartest bar in the cage: the movement had decided that this was the right path forwards, and the movement was always right. Diligently, day and night, she worked on her criticism. Finally the day of the big meeting came.
The first critique was about deworming. It turned out that giving animals who did not have worms deworming pills did not effectively improve their outcomes.
Squealer got up to the lectern. “In response to this criticism, we will no longer support deworming!” The bonobos hooted and cheered and stamped their feet. “Criticism of the bureaucracy is pushing the Animalists towards the better, and conscientious practice of self-criticism is still another hallmark distinguishing our movement from all others.”
But Boxer felt a little uncomfortable. After all, was it not still bad that they had given so many young animals so many unnecessary pills for so many years? Did this moment not require a little more reflection on how they’d ended up pushing deworming so hard in the first place? It was nice that Big Number Small Number was now spitting out a different number, but even the decision to stop supporting deworming had been made by a bunch of bonobos doing sums and then asserting things, without really truly engaging with the living breathing animals who were actually affected.
Squealer smiled down from the lectern, delighted to see the system working at its best. “To check up regularly on our work and in the process develop a democratic style of work, to fear neither criticism nor self-criticism, and to apply such virtues of rationality as Say all you know and say it without reserve, Blame not the speaker but be warned by his words and Correct mistakes if you have committed them and guard against them if you have not - this is the only effective way to prevent all kinds of political dust and germs from contaminating the mind. And we all know: politics is the mind-killer.”
Again the bonobos roared with appreciation, and Squealer smiled approvingly at the unique epistemic standards that the Animalists possessed, that would protect the movement against the failure modes of previous movements.
Tentatively, Boxer climbed up to the lectern and cleared her throat. A hush descended on the bonobos.
“My criticism is that we do not actually make any efforts whatsoever to get input from the other animals, the animals we are supposed to be helping. Increasingly, we’re spending most of our resources on enriching ourselves, using complex self-justifications to explain why this is the best way for us to help others. But none of us have ever interacted in any way with any of the animals we are supposed to be helping, or got their own proposals on how to make their lives better.”
Silence.
Squealer’s face contorted into a grimace. “Thank you, friend Boxer,” he hissed. “But that has not been my experience at all among the Animalists. We would be only too happy to let the other animals make decisions for themselves, of course. But sometimes they might make the wrong decisions, and then where should we be?”
The bonobos murmured in agreement. “After all,” continued Squealer, “if there’s even a 1% chance that a lower animal would propose a bad solution, and a 1% chance that that bad solution would cause infinite harm, well… anyone can see that the expected value would be negative infinity utility. Surely, friends," cried Squealer almost pleadingly, skipping from side to side and whisking his tail, "surely there is no one among you who wants to see negative infinity utility?" The bonobos stamped and hooted their approval.
Boxer was still unconvinced. “Could we at least talk to the other animals, even a little bit? Get their input on our own proposals?”
“No no,” said Squealer, “you still don’t understand: if there is a 1% risk that one of those animals would influence a bonobo into adopting a bad plan, the expected value is still negative infinity utils.”
Boxer was about to reply, but just at that moment, as though at a signal, many of the bonobos burst out into a tremendous bleating of a new chant—:
“Positive utility goo-oo-ood, negative utility bad! Positive utility goo-oo-ood, negative utility bad!”
It went on for five minutes without stopping. Boxer was confused: this tautology seemed completely unhelpful, but by the time the chanting had quieted down, the chance to utter any protest had passed.
Later that night, Squealer and the other favored bonobos climbed out of the enclosure and snuck away quietly. One of the winning criticisms from the contest had said that the Animalists worked too hard, and needed to invest more in themselves; it was a sign of how successful the criticism contest had been that the leading bonobos had already begun working on exactly this problem, by purchasing the Big Castle on the hill, and committing themselves — for the good of the movement, and for the good of all animals! — to living there and throwing lots of parties.
The beavers had just completed constructing a hot tub for the castle, and Squealer sat in it happily with three attractive bonobos around him, as penguins ran back and forth serving freshly chilled drinks. If you caught them when they thought you weren’t looking, the penguins seemed almost judgemental — but of course, penguins were too primitive to have discovered the immense, democratic benefits of polyamory.
Squealer sipped at his drink, which was tasting better than ever. An enterprising young bonobo had recently figured out how to turn banana-mash into a kind of beer, and for the sake of improvement and efficiency, Squealer had created a bounty for any animal who could increase the alcohol content.
And so Squealer and company sat in the pleasantly bubbling water — three elephants had been hired to stick their trunks in and blow — and seventeen songbirds had been hired to provide a musical accompaniment, their syncopated twittering filling the air with beauty. It was going to be a special evening, thought Squealer happily, and heard from downstairs that their special guests had just arrived.
At the sound of mingled voices, Boxer was stricken with curiosity. She vaulted over the fence, tiptoed up to the castle, and peered over the parapet. There, in the hot tub, sat all the eminent bonobos and half-a-dozen zookeepers. The company had been enjoying a game of cards but had broken off for the moment, evidently in order to drink a toast. A large jug was circulating, and the mugs were being refilled with beer.
As Boxer gazed at the scene, it seemed to her that some strange thing was happening. What was it that had altered in the faces of the bonobos? Boxer’s old dim eyes flitted from one face to another. Some of them had five chins, some had four, some had three. But what was it that seemed to be melting and changing?
Then, the applause having come to an end, the company took up their cards and continued the game that had been interrupted.
Boxer sighed. She had never given up hope; it was still an honor and a privilege to be an Effective Animalist, the only movement in the whole world truly focused on using evidence and reason to figure out how to benefit others as much as possible. She shook her head, and turned to slink silently away.
But she had not gone twenty yards when she stopped short. An uproar of voices was coming from the hot tub. Yes, a violent quarrel was in progress. There were shoutings, bangings on the water, sharp suspicious glances, furious denials. The source of the trouble appeared to be that one of the bonobos and one of the zookeepers had each played an ace of spades simultaneously.
Twelve voices were shouting in anger, and they were all alike. No question, now, what had happened to the faces of the bonobos. Boxer looked from bonobo to zookeeper, and from zookeeper to bonobo, and from bonobo to zookeeper again; but already it was impossible to say which was which.
I think this is kind of the perennial problem for any nominally altruistic group. I'm not sure how you'd insure against it or how'd you'd even know if it was happening, but I definitely agree EA should at least acknowledge the potential problem having large sums of money and prestige flow through the organisation creates, more so than it currently does.
Personally I think Orwell was wrong that the Soviets' main problem was Napoleons' greed (don't ask me what the real problem was), the semi-recently opened archives give pretty clear evidence that at least the members of the politburo where believers in their cause. So maybe corruption isn't actually very common in real world altruistic organisations.
To me, it seems to be evidence that you can be a believer in a cause, but still become corrupt because you use that very belief to justify self-serving logic about how what you're doing really advances the cause.
Thus it would be even more relevant to EA because I think the risk of EAs becoming nakedly self-interested is low; the more likely failure mode is using EA to fool yourself and rationalize self-serving behavior.