Re the banning idea, I think you could fall afoul of "unnecessary rudeness or offensiveness", or "other behaviour that interferes with good discourse" (too much volume, too low quality). But I'm not the moderator here.

My point is that when you say that Gwern produces verbose content about a person, it seems fine - indeed quite appropriate - for him to point out that you do too. So it seems  a bit rich for that to be a point of concern for moderators.

I'm not taking any stance on the doxxing dispute itself, funding delays, and so on.

I honestly don't see such a problem with Gwern calling out out Charles' flimsy argument and hypocrisy using an example, be it a part of an external dispute.

On the other hand, I think Charles' uniformly low comment quality should have had him (temporarily) banned long ago (sorry Charles). The material is generally poorly organised, poorly researched, often intentionally provocative, sometimes interspersed with irrelevant images, and high in volume. One gets the impression of an author who holds their reader in contempt.

Although, the costs of insurance would need to be priced according to the ex ante costs, not the ex post costs.

For example: Bob embarks on a project with a 50% chance of success. If it succeeds, it saves one person's life, and Bob sells the IC. If it fails, it kills two people.

Clearly, the insurance needs to be priced to take into account a 50% chance of two deaths. So we would have to require Bob to buy the insurance when he initially embarks on the project (which is a tough ask, given that few currently anticipate selling their impact). Or else we would need to rely on a (centralised) retrospective evaluation of ex ante harm, for every project (which seems laborious).

I take it that Harsimony is proposing for the IC-seller to put up a flexible amount of collateral when they start their project, according to the possible harms.

There are two problems, though:

• This requires centralised prospective estimation of harms for every project. (A  big part of the point of impact certificates is to evaluate things retroactively, and to outsource prospective evaluations to the market, thereby incentivising accuracy in the latter.
• This penalises IC-sellers based on how big their harms initially seem, rather than how big they eventually turn out to be.

It would be better if the IC-seller is required to buy insurance that will pay out the whole cost of the harm, as evaluated retrospectively. In order for the IC-seller to prove that they are willing to be accountable for all harms, they must buy insurance when they sell their IC. And to ensure that the insurer will pay out correctly, we must only allow insurers who use a standard, trusted board of longtermist evaluators to estimate the harms.

This means that a centralised system is only required to provide occasional retrospective evaluations of harm. The task of evaluating harms in prospect is delegated to insurers, similar to the role insurers play in the real world.

(This is my analysis, but the insurance idea was from Stefan.)

My best guess is that without Eliezer, we wouldn't have a culture of [forecasting and predictions]

The timeline doesn't make sense for this version of events at all. Eliezer was uninformed on this topic in 1999, at a time when Robin Hanson had already written about gambling on scientific theories (1990), prediction markets (1996), and other betting-related topics, as you can see from the bibliography of his Futarchy paper (2000).  Before Eliezer wrote his sequences (2006-2009), the Long Now Foundation already had Long Bets (2003), and Tetlock had already written Expert Political Judgment (2005). 

If Eliezer had not written his sequences, forecasting content would have filtered through to the EA community from contacts of Hanson. For instance, through blogging by other GMU economists like Caplan (2009). And of course, through Jason Matheny, who worked at FHI, where Hanson was an affiliate. He ran the ACE project (2010), which led to the science behind Superforecasting, a book that the EA community would certainly have discovered.

like Bostrom's influential Superintelligence - Eliezer with the serial numbers filed off and an Oxford logo added

It's not accurate that the key ideas of Superintelligence came to Bostrom from Eliezer, who originated them. Rather, at least some of the main ideas came to Eliezer from Nick. For instance, in one message from Nick to Eliezer on the Extropians mailing list, dated to Dec 6th 1998, inline quotations show Eliezer arguing that it would be good to allow a superintelligent AI system to choose own its morality. Nick responds that it's possible for an AI system to be highly intelligent without being motivated to act morally. In other words, Nick explains to Eliezer an early version of the orthogonality thesis.

Nick was not lagging behind Eliezer on evaluating the ideal timing of a singularity, either - the same thread reveals that they both had some grasp of the issue. Nick said that the fact that 150,000 people die per day must be contextualised against "the total number of sentiences that have died or may come to live", foreshadowing his piece on Astronomical Waste, that would be published five years later. Eliezer said that having waited billions of years, the probability of a success is more important than any delay of hundreds of years.

These are indeed two of the most-important macrostrategy insights relating to AI. A reasonable guess is that a lot of the big ideas in Superintelligence were discovered by Bostrom. Some surely came from Eliezer and his sequences, or from discussions between the two, and I suppose that some came from other utilitarians and extropians.

Five recruitment ideas.

Here are five ideas, each of which I suspect could improve the flow of EA talent by at least a few percent.

1. A top math professor who takes on students in alignment-relevant topics

A few years ago, this was imperative in CS. Now we have some AIS professors in CS, and a couple in stats, but none in pure math. But some students obsessed with pure math, and interested in AIS, are very bright, yet don't want to drop out of their PhDs. Thus having a top professor could be a good way to catch these people.

2. A new university that could hire people as professors

Because some academics don't want to leave academia.

3. A recruitment ground for politicians. This could involve top law and policy schools, and would be not be explicitly EA -branded.

Because we need more good candidates to support. And some distance between EAs and the politicians we support could help with both epistemic and reputational contamination.

4. Mass scholarships for undergrads at non-elite, non-US high-schools/undergrad, based on testing. This could award thousands of scholarships per year. 

A lot of top scientists study undergrad in their own country, so it would make sense to either fund them to move to a better-connected environment, or to try to reach them in situ. (I think Atlas is higher-touch, and pitched younger than this would be.)

5. Internship & scholarship programs for German and Australian medical programs to transition into biosecurity.

Germany and Australia give school-leavers a singular score that determines what university programs they can enter, and medical programs are the most competitive, so a ton of EAs from those countries are medical undergrads. We could lean into this. Probably some other countries are the same.

AI Seems a Lot More Risky Than Biotech

We tend to think that AI x-risk is mostly from accidents because well, few people are omnicidal, and alignment is hard, so an accident is more likely. We tend to think that in bio, on the other hand, it would be very hard for a natural or accidental event to cause the extinction of all humanity. But the arguments we use for AI ought to also imply that the risks from intentional use of biotech are quite slim.

We can state this argument more formally using three premises:

1. The risk of accidental bio-x-catastrophe is much lower than that of non-accidental x-catastrophe.
2. A non-accidental AI x-catastrophe is at least as likely as a non-accidental bio x-catastrophe.
3. >90% of AI x-risk comes from an accident.

It follows from (1-3) that x-risk from AI is >10x larger than that of biotech. We ought to believe that (1) and (3) are true for reasons given in the first paragraph. (2) is, in my opinion, a topic too fraught with infohazards to be fit for public debate. That said, it seems plausible due to AI being generally more powerful than biotech. So I  lean toward thinking the conclusion is correct.

In The Precipice, the risk from AI was rated as merely 3x greater. But if the difference is >10x, then almost all longtermists who are not much more competent in bio than in AI should prefer to work on AIS.

Without the timestamps:

This is Sommar in P1 with Nick Bostrom.  

The good thing about bird poo is that it’s small: that’s how a philosopher thinks. Other people are hit by a bird poo and they think “That wasn’t good”. I get a bird poo on me and I think“It might be a lucky sign. I got a bird poo on me but all of the other poos missed me. THAT is happiness.” My name is Nick Bostrom, and I am a philosopher. Welcome to my “Summer in P1” from Oxford where I live. I grew up in Helsingborg, just next to the beach by Öresund. It was a safe and happy childhood, but there was one worm in the apple. One big, disgusting, repulsive worm: School. As I saw it, I could already read, write and count way before I started school and what they wanted to teach me was just the name of a lot of rivers. Was it really worth investing half of one’s childhood to learn the names of some rivers and counties? I doubted that. My strategy in school became doing as little as possible. When we approached the end of middle school [age ~10-12], and were going to start seventh grade and high school [age ~13-15], we all got to have a meeting with the teacher, one on one. The idea was probably for us to receive some encouragement and, perhaps, some good advice and words of wisdom, before the rest of our lives. We were sitting there in the room, and the teacher said to me: “You will fail”. I continued to be unhappy when I started high school after the summer vacation. But a year or two later, something strange happened. It was one day, I guess in year eight, or maybe nine. I was walking around by myself without anything in particular to do. It occurred to me to go to the city library of Helsingborg. Why, I don’t remember. In my family there were no academics, and I associated everything that had to do with books with school, which I despised. In any case, I was walking around those halls, picking from the shelves. I started browsing a book which turned out to be an anthology with different German philosophers from the 19th century. I think one of the names was Schopenhauer. Nietzsche was probably in it too. In any case, I borrowed that, and some other books, and brought them home with me. I brought the book with the German 19th century philosophers to my grandma and grandad who lived close to us. They had a garden. And so, I sat down beneath an apple tree and started reading. The world that opened up to me, there under the apple tree, resembled nothing I had experienced before. I read more and more from the book, and it was as if a big door opened up to a world of ideas and thoughts which I, up until that point, had been completely unaware of my whole life. In the time that followed, I found my way, in quick procession, to philosophy, science, art, and literature. I found that when I read texts which were considered difficult, they often seemed completely clear and understandable to me. I started reading them in their original languages, which was even better. I was around 15 years old, but it felt as if I had slept my way through life, up until this point, and only now had I opened my eyes. I was struck by a strong feeling that I had wasted my time up until this awakening and that, now, I didn’t want to lose even one more minute. This was the start of my intellectual journey. I cast myself into my own education project with all my energy, and tried to master lots of different areas; from reading scientific papers, to writing poems and plays, and, above all; I started thinking. "Okay, so there’s a universe which started existing, and here we are on planet Earth, in Skåne, out of all places. And we can see that there are at least a billion billion other solar systems out there in space - many of them much older than our solar system. And the modern age has just begun for us, after having been hunters and gatherers for a couple of hundred thousand years. And here, there’s a little boy called Niklas Boström beneath an apple tree, and he’s got a cranium, and in that cranium there’s a brain which weighs 1,5 kilos. And everything I see, and hear, and think are electrical processes inside that brain, which will die in a few decades, or so. Hm, hm, hm, hmm...” A feeling had been born within me that time constantly flows like sand through our fingers and disappears. We have one life, a time-limited opportunity to make of that life something that counts. And to perhaps try and understand what it’s all about. 

One of the many things I did during this time was painting. I built myself a studio, not far away from the apple tree, in my grandparent’s basement. More specifically, in their laundry room. Down there, more and more paintings were created. Oil painting and drawings. I don’t know how many times my grandma found paint stains on her newly washed sheets. But she didn’t complain once. Near their house is Pålsjö forest, which looks out on the beach of Öresund. I had the idea to put up an exhibition there in spring, as the leaves started blooming, and the ground was covered with wind-flowers. I cycled around town, putting up posters about this big art exhibition which was to be held. I borrowed a garden cart, and a wheel-barrow from my grandad. And with one in each hand, I rolled out a selection of my paintings to a glade in the forest. I hung my paintings directly on the trees. The exhibition would be held for four days. Day 1: Pouring rain. The wheels of the garden cart and the wheel-barrow dig down into the mud. Day 2: Pouring rain. Day 3: Pouring rain. Day 4: Just normal rain. The next year I did it all again with a new exhibition. It rained again. The wooden frames of the paintings were twisted in the wetness, like tormented souls. Regardless, I had managed to sell a few paintings. Helsingborgs Dagblad [daily newspaper] came by and wrote a paragraph about it, but shortly thereafter I stopped painting. There were other things to put that time and effort into. Things that I thought were more important. If you objectively consider what the world is in most need of, it is not very likely that the correct answer is more oil paintings. 

During this time, I used to get on my bike after school and cycle out into the forest. Out there, I read my books, wrote poems and aphorisms and plays, and thought about philosophy. And out there, I experienced high levels of happiness. Often, I felt pierced by the beauty of nature.I could sit there, my gaze fixed, and simply stare at a scene or some object which caught my attention for 10, 20 or 30 minutes. It could happen elsewhere, as well. Sometimes it wasn’t a physical object or a scene, but a philosophical contemplation. It was as if my consciousness was filled by an idea to the extent that my own self disappeared temporarily. A pure observation, a pure contemplation of something in itself, without reference to its usefulness or conventional meaning. Now, one might think that this leap into the world of ideas and literature would make me more interested in school, but it actually turned out the other way around. If, earlier, school had been boring, I now found it to be an unbearable waste of time. I felt that I could develop quicker on my own, at home or in the forest, than in school, behind my bench. And this became an increasingly big problem for me. In high school/sixth form [ages ~16-18], I found the solution to this problem. I found a way, which most times was only used by students who had been away for an exchange year. If you had learnt a subject, you could be examined in it through special exams, and then you wouldn’t have to go to the class. It turned out that there was no rule against doing this in all subjects. I could hardly believe it. Everyone around me seemed to be of the opinion that it was a bad idea and they dissuaded me, but I decided to stay home from school. And thus I managed to be examined in the natural sciences programme, already before the start of my last year, subject by subject. It took me 10 weeks, which I thought confirmed that school had really just been a big waste of time. But now, as I was free from school, a new problem arose. The call to enlist in military service. The last thing I wanted, after having freed myself from the school system, was to end up in another state institution. I remember sitting there in the waiting room at the enlistment centre and reading McClelland, Rumelheart, and the PDF Research Groups’ groundbreaking "Parallel Distributed Processing, Volume 2: Explorations in the Microstructure of Cognition: Psychological and Biological Models". Next to me sat some guy who was playing with his lighter, almost setting fire to the shirt of one of the other people who had been called in. So, I did the different tests for cardio, arm strength, leg strength, hand strength, sight, hearing, and intelligence. At last, it was time for a meeting with the official who would assess how I would do mentally in the military. He asked about my interests. “I like sitting on a rock in the forest, writing poems”. He nodded and asked: “But, if there was a war, then what would you do?”. "Well, I could write poems about how horrible the war is”, I replied. "Yes, someone has to do that too”, the man responded. And I received my exemption warrant. 

When freedom was finally mine, I saw two different directions I could go in. The first was to move to a shed in the forest somewhere and keep doing the kind of thing that I was already doing. Fortunately, I didn’t choose that path. The other one was university. I was hesitant after the dull years in school, but I decided to at least try. It turned out that university was something completely different from school. So I dove into it; hook, line and sinker. After a short while in Lund, I moved to Gothenburg. I studied mathematics, philosophy, mathematical logic, and artificial intelligence, at the same time as I kept reading a lot on my own. The feeling that there was no time to lose, I still carried with me. I’ve always been a night owl, and need a lot of sleep, so I ran to lectures in the morning, while shoving down a cheese sandwich, which I had prepared the night before. No time would go to waste. Around this time, I sometimes kept a bucket of cold water in my room. When I got tired and couldn’t concentrate, I put my head into the bucket to try and wake up. It didn’t work particularly well. Anyway, I felt restless in Gothenburg, and after a year or so I was recommended by an acquaintance to move to Umeå, where the university was supposed to be good. I also started studying psychology there, among other things. After two or three months I was called in for meeting in the rector's/head of department office at the psychology department. An official there told me that he had discovered that on the side of psychology, I was doing three other full time degrees; physics, mathematics, and philosophy. (He hadn’t found out that I was also studying logic remotely.) He told me that I had to quit the other degrees, or he would throw me out from the psychology department, on account of it being psychologically impossible to study all of that at once. I tried telling him: "You know, so far it has worked fine”. I stood there with the certificates showing that I had pretty good marks in all of the courses. Wouldn’t it be better to try and enthuse students who did not have enough motivation, than drive out the ones who really had a hunger for learning? No, I was expelled. Perhaps the first person to ever be expelled from an education establishment for studying too much. In any case, I didn’t mind much. This was in the autumn term, and I had noticed that each week in Umeå [in northern Sweden] was getting colder and darker than the one before. I could see where we were heading. I was happy to return to Gothenburg. Throughout this whole period, which lasted six or seven years, I carried with me a feeling of deep, deep loneliness and isolation. All my life I had never met even one person who shared my interests. I felt as if of an alien species, who had, in some way, landed on this planet by accident. Despite this feeling of isolation, or perhaps because of it, I also carried a feeling of deep empathy for all that humans, and animals for that matter, have to endure in their lives. All our vulnerability in the face of the dark forces of the universe and our existence, against which we are still today mostly defenceless, however much modern technology we have. These thoughts could perhaps be traced back to everything I had read of the German philosophers, and the Swedish poet and bishop Esaias Tegnér. In fact, I had read so much Tegnér and other Swedish classics that I started speaking with old Swedish grammar, with verbs in plural conjugations; “The frogs aren funny to observe, for they haven no tails”.* *[Note: This sentence is a reference to a very common midsummer song; "Små grodorna", which is sung while dancing around the Maypole, imitating frogs. (The actual song is not sung with these old words in it, but in modern Swedish.). ] To speak in this manner probably has to be regarded as one of the least good ideas I’ve ever had; even if I myself thought it sounded very good at the time. In any case, it hardly helped to reduce my isolation. Strangely enough, no one ever commented on how I spoke. People probably assumed that I was speaking with some particularly wacky Skåneian accent. And since then, even if I now have stopped speaking with an old Swedish accent, my Swedish has worsened. I’ve lived abroad now for so long that I’ve almost lost my ability to speak the mother tongue. So, that’s why it sounds a bit strange. And I’ll have to ask you to have some ‘patience’ with this. 

After a year, or so, I moved to Stockholm, where I started studying theoretical physics and philosophy. At this time I was just about to undergo a new transformation. Almost as profound as in my teenage years when I discovered the world of ideas. One could perhaps describe it as a transition from a romantic to a modern epoch. I lost my interest in the 19th century philosophers and in reading classics, and in the heroism of the past. Instead, I got interested in analytic philosophy and modern science; by the potential in technology. You could say that in the step from past to present I missed the mark and flew past, and became more interested in the future instead. This was an enormous shift in my focus. I was done with writing poetry and such things. The last poem I wrote was called ‘Requiem’, and it is actually an attempt to describe this mental transformation. It is about a general in the cavalry who is lost, and wakes up and sees his camp abandoned. It takes us on a compressed journey from an old heroic era, to an entirely modern, or even postmodern, era. I’ll read it to you now. If you need to go to the laundry, or get something else done, this is a very good moment. När tidens tupp för tredje gången gal, ur hjälteslummern väcks vår general. ( When time’s rooster for the third time crows, out of his heroic slumber our general is woken up. ) Han står upp ( He stands up ) det är för sent: ( it is too late: ) lägret ligger övergivet; ( the camp is abandoned; ) hans trupp har dragit å färde. ( his troop has moved on. ) Det är kallt; ( It is cold; ) gräset hans stövlar träder på är stelt av den första frosten ( the grass his boots tread on is stiff from the first frost ) De djupt insjunkna ögonen vittnar om utståndet lidande. ( His deeply sunken eyes testify to endured suffering. ) Anletsdragen, som åren utmejslat, bär prägel av ansvarsfullhet. ( His facial features, which the years have chiseled, bear the imprint of responsibility. ) Han vandrar stilla runt bland tälten med manteln över axlarna, tigande. ( He wanders quietly around the tents, cloak over his shoulders, silently. ) Det är ingen där. ( There is no one there. ) Han verkar överlägga något. ( He seems to deliberate over something. ) Så går han raskt fram till sitt riddjur, ( Then, he walks towards his mount, rapidly, ) sätter foten i stigbygeln och svingar sig upp. ( places his foot in the stirrup and swings himself up. ) Han smackar åt hästen och rider iväg utan ord. ( He clicks, and the horse rides off, without words. ) Sporra ditt riddjur! Kanske är det inte försent ännu, att samla styrkorna, ( Spur your mount! Perhaps it is still not too late, to gather the forces. ) att i spetsen för din här segra för humaniteten! (smack! smack!) ( to at the head of your army prevail for humanity! (Click! Click!) ) Hoppar över röset, plaskar genom ån, rider under lundarnas bokar ( Leaps over the cairn, splashes through the stream, rides under the beeches of the groves, ) och ut över fälten, där morgondimman ligger drömmande kvar, och förbi kärret, ( and out on the fields, where the morning fog dreamingly remains, and beyond the swamp, ) där ljudsalvorna från en hackspetts knackande hamrar bort över ytplanet. ( where the noise from a woodpecker’s rapping is hammering away over the surface. ) Hans vilja tvingar fram oanade krafter ur hästens väsen. ( His will forces unsuspected strengths out of the horse ) Han tar sig upp på åsen. ( He makes his way up on the ridge. ) Piskrappen går som elektricitet genom nerverna, ( The whip lashes pass through the nerves like electricity, ) får den svettglänsande muskulaturen att uträtta arbete; ( get the sweat glistening musculature to do the work; ) och sporrarnas blodade järngaddar tillhandahåller en smärta ( and the spurs’ blooded iron stings provide the pain ) som stegras till det outhärdliga om steget mattas det minsta. ( which intensifies to the unbearable if the step slows down even just a little. ) Fradga bildas från djurets gipa, dess ögon är vilda ( Froth appears from the corner of the horse’s mouth, its eyes are wild ) näsborrarna är vilda. ( its nostrils are wild. ) Då hörs i den galna galoppen ett dån, större än åska. ( Then, in the mad gallop, a roar louder than thunder is heard. ) Hjälten greppar hårt om värjskaftet, ( The hero grips the sword handle tightly, ) vänder trotsigt blicken uppåt. Maktens musik och himmelns trafik ( turns his gaze upward defiantly. The music of power, and the traffic of the sky, ) ur skyn skjuter ett jaktplan! ( out of the sky shoots a fighter jet! ) (Men bakom gudens styrpanel sitter en människoapa.) ( (But behind the god’s dashboard, sits a human ape.) ) Fåglarna flyr i samtliga vädersträck ( Birds are fleeing in all directions ) som om denna örnen jagade dem. ( as if this eagle were hunting them. ) Ljusstrålar som reflekteras från föremålet ( Light rays reflected by the object ) bryts samman av linsen och fokuseras på näthinnan, där fotoreceptorer stimuleras att avge aktionspotentialer ( are refracted by the lens, and focused on the retina, where photoreceptors are stimulated to emit action potentials ) som fortplantas utefter nervus opticus ( which are propagated along nervus opticus, ) och projiceras på synbarken, som sitter här ( and projected on the visual cortex, which is located here ) och framför den sitter själen — uppochnedvänd i tevefåtöljen och glor och käkar popkorn. (Wow! vad det är spännande!) ( And in front of it, is the soul - upside-down in the armchair by the TV, staring and eating popcorn. (Wow! It’s so exciting!) ) Emellertid avlägsnar sig det snabba flygplanet snart; ( However, the high-speed aircraft soon departs; ) vår krigman står bredvid sin fåle och följer med blicken ( our war man is standing next to his steed, and follows the dot with his gaze ) pricken över horisonten som småningom försvinner helt i det blå.. ( over the horizon, and soon it disappears completely into the blue… ) Reformatorn står nedstänkt i renen: passerad av novationer dem han själv installerat. ( The reformator stands there, stained in the rail: passed by novations he has installed himself. ) Ångan flåsar från deras munnar som ur två föråldrade lokomotiv... ( The steam puffs out of their mouths as from two obsolete locomotives. ) Stannar tiden, om vi står alldeles stilla? ( Will time stop, if we stand completely still? ) Perplex är den vise; så det blir hans häst som skiter fram propositionen ( Perplex is the wise, so it’s his horse who shits forward the proposition ) att omdefiniera våra ideal. ( to redefine our ideals. ) "Hä, även du min Brunte* även jag. ( “Ha, even you, my Brunte* even I. ) *[Note: Typical horse name in Swedish] Är alltså min art dömd till undergång, som din till att äta hö?" ( So, is my species then condemned to destruction, as yours is to eating hay?” ) Slutligen ger sig dock saken av sig själv, ( Eventually, however, the thing is resolved by itself, ) faller ner från himlen som en rostad duva på tallriken. ( falls down from the sky like a roasted dove on the plate. ) Sticka vapnet i skidan; bege sig hem; äta frukost; läsa tidningen. ( Put the weapon in the scabbard; go home; eat breakfast; read the newspaper.) Brigadgeneralen, som städse gått mot faran utan att blinka, blundar icke nu för att hans yrkesverksamma liv är över; ( The brigadier, who has often stood up against the danger without blinking, will not shut his eyes now just because his professional life is over; ) han kan under alla omständigheter påräkna pension från staten. ( in any case, he can count on his pension from the state. ) Kanske är det bäst så. Han till och med visslar en melodi. ( Perhaps that’s for the best. He even whistles a melody. ) Då kommer det fram en råtta, vit med röda ögon, och säger: ( Then, a rat appears, white with red eyes and says; ) "Vad är Mozarts musik mot stimulering av nucleus accumbens." ( "What is the music of Mozart compared to stimulation of the nucleus accumbens” ) Befälet sparkar undan råttan. Han överlämnar kommandot åt högre existensformer: optimerade samhällssystem, artificiella neurala nätverk och medicinerade övermänniskor. ( The commander kicks away the rat. He hands over the command to higher forms of existence: optimised social systems, artificial neural networks and medicated superhumans. ) Lux aeterna luceat eis, Domine: Cum Sanctis tuis in aeternum: quia pius es.

Sometimes when you’re outdoors, you can feel a drop hitting your head. (If you’re a little bald anyway, as I have become with age.) And then you hope that it’ll start raining - because the alternative, if it wasn’t a raindrop, is bird poo. In January of 1996 I came to London. At first, the idea was to stay only for one exchange term at King’s College where I studied computational neuroscience. But when the end was approaching, I felt that I wanted to stay in that stimulating environment and pursue my PhD there. The question was; in what? I was interested in theoretical physics, and in brain science. But I thought that if I did a PhD in one of those subjects, then I’d eventually end up knowing a great deal about something very limited. About the top quark, or about some region in the hippocampus. Instead, I decided to do my PhD in philosophy of science and the foundations of probability theory. In that way, I’d get more of an overview of the big construction work of science, and I could pick out the most interesting parts from different areas. At this time, I had become a little more social. A little more mature. I still studied and worked hard, but I no longer ran to my lectures with a sandwich in my mouth. London, in the mid-90s was, as I said, an exciting place. Those were the years of the short period of Cool Britannia, with Britpop and a flourishing club scene for house music, and new art, and more. People knew basically two things about Sweden: ABBA and Ingmar Bergman. At each and every party in the mid-90’s, ‘Dancing Queen’ was played. And perhaps the ghost of Bergman was present in the tourist book about Sweden that I found at one point. Among phrases such as “What time is it?”, and “How can I get to the train station?”, there was also the phrase; “Think about death”, as one of the necessary bits of knowledge for being a tourist in Sweden. At this time, I also became more involved in activities outside of my degree; in theatre courses and debating competitions. I also did stand-up comedy, a scene and genre which was very lively at the pubs and theatres of London. One thing I liked about stand-up was the honesty and the directness in the feedback from the audience. In stand-up, it’s simply the case that if the audience laughs, then it’s good, and if the audience doesn’t laugh, then it’s crap. No ambiguity. It’s in stark contrast to philosophy, where you can keep going for years and perhaps a whole life without ever definitively knowing whether you’re good or not.

The step into the world of London for me also coincided with the step out on the internet. See, I’m so horribly old that I grew up without the internet. When I arrived in London in 1996, I finally got online, and for me that meant a lot. Online, I discovered that there are other people in the world who are interestedin the questions that I had been thinking about for many years. I had long had the view that the prospects for humanity one day would change fundamentally, as a consequence of the development in artificial intelligence, or that biomedical methods would be able to raise the cognitive ability of humans, or to manipulate human brains to change moods. But I had never met anyone else who had been at all interested in questions like these. Online, I quickly found a community who called themselves transhumanists, or extropians, and who had discussions about these questions. That I happened to stumble across this, was actually the result of the work by another Swede, who’s name is Anders Sandberg, and who today is a research colleague at my institute. Anders had a big website with a lot of links, which I found and started following, and I quickly got involved in the discussion groups and mailing lists. It was a rather unique intellectual environment for that time. It was, admittedly, loud and there was a lot of nonsense and crazy ideas, but there were also some interesting ideas. And the people who participated were often outside the middle lane. There were some academics and some who worked as software engineers, but also people who lived all kinds of different lives and who devoted themselves to thinking about how AI, nanotechnology, space technology, virtual reality, human enhancement medicine, life extension, and so on, could end up affecting human existence in the future. By this time, this was practically the only forum where discussions like this were held. But my view was that, while conversations on the internet admittedly were much better than nothing, nevertheless they were also significantly worse than what they should be. There were two big limitations, as I saw it; Firstly, it was first and foremost a pastime for amateurs who discussed these important questions in their spare time. From my perspective, these questions deserved the same serious and systematic patience and attention which is automatically given by researchers to all sorts of less important areas. A few years ago, we conducted a study at my institute which showed that there’s 10 times more research done on dung beetles than existential risks and the future of humanity. So that had to change. The whole discourse needed to be raised up to a level of serious academic research. The second problem with this early version of transhumanism, was that it all too often lapsed into a kind of simple-minded cheering section for the development of technology. It made the sloppy assumption that technology by itself would for certain be a good force in the world. In my eyes, this seems like a limited perspective. My position was that; Yes, it is reasonable that technology will at some point in the future fundamentally change the nature and prospects of humanity, even if there’s some uncertainty about when this will happen. And yes, there’s also an enormous upside and a potential to create something which is far more wonderful and humane than the world we see around us today. Not only by controlling the outside world, but also by changing and upgrading human nature. But at the same time, there was no guarantee that the results of increasing our technological powers would be good. It would perhaps be just as likely that these new capabilities would come to be used for destructive purposes, and cause catastrophes with effects which are irreversible for the human race and the planet. In my research I have called these ‘existential risks’. The effort should be focused on understanding these different ways that the development of technology can change the world, and find the points where one, with some good will, can increase the probability that this development will be beneficial for humanity. To make it possible to think in these other ways, I founded the World Transhumanist Association together with an acquaintance, David Pearce, around 1997. It was an international grassroots organisation for facilitating discourse about these things, and for making it more academically respected. WTA quickly grew to several thousands of members, and organised conferences, and had a scientific journal. And I think it played a significant role for a few years, by being the forum where these discussions could be held. But I have myself left transhumanism long ago. Already in the early 2000s, we had managed to move these questions into the academic world. Improvement of the human race was discussed as a part of bioethics, for example; and another, even more interesting, branch led to the work that now has been continued by, for example, The Future of Humanity Institute, about which I will say more in a moment.

After my PhD in London, I moved to the US and to Yale University. During my time at Yale, I published a scientific paper which introduced the simulation argument. I will not go into any kind of I won’t give a lecture about it, but will only read the abstract of the article. The title was “Are You Living In a Computer Simulation?” “This paper argues that at least one of the following propositions is true: (1) the human species is very likely to go extinct before reaching a “posthuman” stage; (2) any posthuman civilization is extremely unlikely to run a significant number of simulations of their evolutionary history (or variations thereof); (3) we are almost certainly living in a computer simulation. It follows that the belief that there is a significant chance that we will one day become posthumans who run ancestor‐simulations is false, unless we are currently living in a simulation. A number of other consequences of this result are also discussed.” The simulation argument had a big influence when it was first published in Philosophical Quarterly, and has continued to have a big influence ever since. It seems like the attention it gets comes and goes in waves; when some new community discovers it for the first time. The simulation argument has also appeared in popular culture. The founder of Tesla, Elon Musk, repeatedly tweets about it, and there are several films and short stories, music, and a production on Broadway, which have been directly inspired by it. Right now, there’s apparently someone who’s in the process of putting together a ballet about this academic article in Volume 23, Number 211, of Philosophical Quarterly, 2003. 

In 2003 I came back to Europe, to Oxford University. After a few years, I had the opportunity to found The Future of Humanity Institute. The institute originally came into being as a consequence of a big donation to the university by the philanthropist Dr. James Martin. He wanted to support interdisciplinary work with a focus on humanity’s challenges in the 21st century. FHI started as one of ten institutes in this investment within the University of Oxford. Yes, the name ‘The Future of Humanity Institute’ sounds quite bombastic. But it is actually also a pretty relevant name. It describes the issues we are working with. The big picture. The major issues concerning human civilization and the survival of the human species in a future where technology could fundamentally change all parameters and conditions of the life we ​​live. At the institute, we try to shape concepts and create intellectual tools that help us to think systematically about these issues. What the big lines are, what opportunities there are for those who want to try to make the world better. How can long-term results for humanity be linked to concrete actions and projects today? The name is also wide enough to give us a lot of room for manoeuvring. One of the things that has arisen from the work at the institute is the book “Superintelligence: Paths, Dangers, Strategies”, which I released in 2014. And it was published in Swedish in 2017. The book took me six years of intense work to write. Initially, the idea was actually for it to be a book about existential risks in general, and that one chapter only would be about artificial intelligence. But when I started writing that chapter, it started growing and growing and growing. It was like a book trying to get itself written, but it was about AI, artificial intelligence, in the future. So I switched focus and wrote the whole book about that. Strangely enough, despite the book being written through a purely academic perspective, a tonne of footnotes, and a pretty heavy prose, it ended up as a New York Times bestseller. It deals with, among other things, what happens if and when us humans succeed in creating general artificial intelligence - that is, not only AI which is better than humans at some specific things, but which outdo the human brain in general problem solving ability and in learning ability. In other words, about what happens when we get superintelligence. With the emergence of superintelligence, we stand before a significant problem. How can one program superintelligent intellects in such a way that they do what we want them to do? That they are safe, and have a good effect on humanity? When the book was released this was a very overlooked area, but during the last couple of years, Perhaps as a consequence of the book, this has changed. Now, there’s an active research field which is attempting to develop methods for controlling AI that will keep working even if artificial intelligence in the future reaches superhuman levels of thinking ability. At FHI we have a group who does research on this today, and which cooperates closely with DeepMind and other leading actors within the field of AI.

Since the start, FHI has grown. We are now approaching 100 employees. When you walk through FHI today it, at first sight, looks very much like a normal academic research centre; offices, meeting rooms, whiteboards with mathematical symbols. But if you stay around for a moment, I think you’ll notice a certain atmosphere; a palpable feeling of purpose. No matter the time of the day, or day in the week, there’s almost always someone there. And practically everyone who works there goes to work, I think, not first and foremost to get a salary, but because they feel that what they do is meaningful. It’s a very special environment. We have a few different research groups. One is focused on artificial intelligence and on developing control methods which could work even if we get superintelligence in the future. Another group, political scientists and such, study how society would work if you have AI which could do all kinds of things which humans now do. And we have another group which works on biotechnology and biosecurity. Macrostrategy is a continuingly big area. Me, I’m currently thinking about the moral status of digital intellect. If you think that we’ll develop artificial intelligence, perhaps in the near future, who will have similar capacities as animals, like a mouse or a cat, and we believe that animals have a certain moral status (you cannot do whatever you want; you cannot kick a dog), so how should we think if our computer programs perhaps gain consciousness ,or the ability to feel pain, and what does that mean, morally speaking? The Future of Humanity Institute is only a small part of a bigger global network which has grown out of what we, and others, have done. Another node in this network is the Future of Life Institute, which was created by another Swede, Max Tegmark, in the Boston region. Actually, there have been surprisingly many Swedes involved in this. There was a time, early in the history of FHI, when we were still very small, with only a handful of employees, when by a coincidence three of us were Swedish. My big fear at this time was that when we’d be advertising a new research job, the most qualified person would be another Swede, because in that case I would have had to employ that person, and to the external world that would look incredibly nepotistic, as it would be statistically unlikely that we’d be so many from my homeland by a coincidence. Luckily enough, the proportions have balanced out since then. Many of those who are involved in this are in themselves remarkable people. For example, one guy who came here to help FHI a year or so ago was so dedicated to the goal of doing as much good in the world as possible, that he not only had donated all of his money to charitable causes, but also donated one of his kidneys. Another, completely wonderful, person was a student who tried to donate as much of his income to charity as possible. So, to save more money, he lived in a tent on campus. Since then, he has had a successful career within the American defence establishment where he hopes to be able to push things into a more humanistic and humane direction eventually. Intellectually speaking, as well, FHI is a special place. We have researchers from a variety of areas. We’ve got several mathematicians, computer scientists, AI-researchers, as well as some political scientists, engineers, and philosophers. Several of these are versatile and bring together several different disciplines. To me, it’s a great privilege to go to work day after day. When I was younger, I’d often read while I was strolling. One of my researchers told me that when he was young, he’d read while he was swimming. I no longer felt like an extraterrestrial who had crash landed on this planet. Or, at least, it seemed like some other extraterrestrials had also survived the crash.

We’re approaching the end of this programme. And this is probably a good place to say thanks to some important people. I’d like to thank my parents for having supported me a hundred percent throughout this wacky journey. I’d like to thank all of my colleagues from whom I have learnt so much, and my wife for all that she has done. And our little boy, for being such a wonderful little being. Well, if one thinks about it, one would like to extend this thanks to hundreds of earlier generations, and hundreds of millions of women and men, whose hard work, courage, self-sacrifice, and creative ways of solving problems have built our modern civilization, whose prosperity and freedom is without equal. We have come a long way, and perhaps we only need to carry the baton a little bit further, until we can hand it over to artificial intelligence, or an upgraded version of humanity. We really need all the help we can get. I’ll end this programme by subjecting you to another poem. This one takes us back to a time when I was a student in Stockholm. It was one winter, during a cold night. I had been at Fysikum [The department of Physics at Stockholm university], analysing research results until late. And I took the underground train home to my student accommodation in Solna. I remember walking from the station through a grove. It was snowing. It was so quiet. The snow suppressed all sounds, and big snowflakes were falling down calmly and peacefully from the night sky. And once again, I was pierced, and I stood there in the night, and just took in the beauty and the grandeur of it. And when, at last, I continued, as in a trance, I wrote this: Stilla faller snön på gatorna och torgen och på den frusna sjön, på glädjen och på sorgen. (ENG: Silently, the snow is falling on the streets and the squares and on the frozen lack, on happiness and sorrow) Singlar sakta ner på svalnade förälskelser. Lägger sig på de eldiga lockar, ja till slut på det flammande hjärtat, de vita flockar. (ENG: Floating slowly down on cooled infatuations, Lays (itself) down on the burning locks, yes at last on the flaming heart, the white flocks. ) Och varje fotspår suddas ut och alla ojämnheter täckas över. Vi hör vid våra dagars slut hur dödens vaggsång barnet söver. (ENG: And every footprint is erased, and all irregularities be covered. We hear, at the end of our days, how the lullaby of death puts the child to sleep. ) Men vak en stund och se flingorna falla som människosjälar ur kosmos det kalla. Det snöar på järnek, lav, och en, Snart snöar det även på din gravsten. (ENG: But wake up for a while, and see the flakes fall like human souls from cosmos, the cold. It is snowing on hollow, lichen, juniper. Soon it is snowing on your gravestone too. ) My name is Nick Bostrom. Thank you for listening.

I agree it's reasonable to ask where (if anywhere) EA is paying too much, and that UK EA has been offering high salaries to junior ops talent. But even then, there are some good reasons for it, so it's not obvious to me that this is excessive.

One hypothesis would be that some EA orgs are in-general overpaying junior staff, relative to executive staff, due to being "nice". But that, really, is pure speculation.