To be fair to Richard, there is a difference between a) stating your own personal probability in time of perils and b) making clear that for long-termist arguments to fail solely because they rely on time of perils, you need it to have  extremely low probability, not just low, at least if you accept the expected value theory and subjective probability estimates can legitimately be applied at all here, as you seemed to be doing for the sake of making an internal critique. I took it to be the latter that Richard was complaining your paper doesn't do. 

How strong do you think your evidence is for most readers of philosophy papers think the claim that X-risk is currently high, but will go permanently very low" is extremely implausible? If you asked me to guess I'd say most people's reaction would be more like "I've no idea how plausible this is, other than definitely quite unlikely", which is very different, but I have no experience with reviewers here. 

I am a bit -not necessarily entirely-skeptical of the "everyone really knows EA work outside development and animal welfare is trash" vibe of your post. I don't doubt a lot of people do think that in professional philosophy. But at the same time, Nick Bostrom is more highly cited than virtually any reviewer you will have encountered. Long-termist moral philosophy turns up in leading journals constantly. One of the people you critiqued in your very good paper attacking arguments for the singularity is Dave Chalmers, and you literally don't get more professionally distinguished in analytic philosophy than Dave. Your stuff criticizing long-termism seems to have made it into top journals too when I checked, which indicates there certainly are people who think it is not too silly to be worth refuting: https://www.dthorstad.com/papers

Fair point, when I re-checked the paper, it doesn't clearly and explicitly display knowledge of the point you are making. I still highly doubt that Thorstad really misunderstands it though. I think he was probably just not being super-careful. 

I am far from sure that Thorstad is wrong that time of perils should be assigned ultra-low probability. (I do suspect he is wrong, but this stuff is extremely hard to assess.) But in my view there are multiple pretty obvious reasons why "time of Carols" is a poor analogy to "time of perils":

1. "Time of carols" is just way more specific, in a bad way than time of perils. I know that there are indefinitely many ways time of carols could happen if you get really fine-grained, but it nonetheless, intuitively, there is in some sense way more significantly different paths "X-risk could briefly be high then very low" than "everyone is physically tied up and made to listen to carols". To me it's like comparing "there will be cars on Mars in 2120" to "there will be a humanoid crate-stacking robot on Mars in 2120 that  is nicknamed Carol".
2. Actually, longtermists argue for the "current X-risk is high" claim, making Thorstad's point that lots of things should get ultra-low prior probability is not particularly relevant to that half of the time of perils hypothesis. In comparison, no one argues for time of carols. 

3. (Most important disanalogy in my view.) The second half of time of perils, that x-risk will go very low for a long-time, is plausibly something that many people will consider desirable, and might therefore aim for. People are even more likely to aim for related goals like "not have massive disasters while I am alive." This is plausibly a pretty stable feature of human motivation that has a fair chance of lasting millions of years; humans generally don't want humans to die. In comparison there's little reason to think decent numbers of people will always desire time of carols.

   4. Maybe this isn't  an independent point from 1., but I actually do think it is relevant that "time of carols" just seems very silly to everyone as soon as they hear it, and time of perils does not. I think we should give some weight to people's gut reactions here. 

Obviously David, as a highly trained moral philosopher with years of engagement with EA understands how expected value works though. I think the dispute must really be about whether to assign time of perils very low credence. (A dispute where I would probably side with you if "very low" is below say 1 in 10,000). 

I think my basic reaction here is that longtermism is importantly correct about the central goal of EA if there are longtermist interventions that are actionable, promising and genuinely longtermist in the weak sense of "better than any other causes because of long-term effects", even if there are zero examples of LT interventions that meet the "novelty" criteria, or lack some significant near-term benefits. 

Firstly, I'd distinguish here between longtermism as a research program, and longtermism as a position about what causes should be prioritized right now by people doing direct work. At most criticisms about novelty seem relevant to evaluating the research program, and deciding whether to fund more research into longtermism itself. I feel like they should be mostly irrelevant to people actually doing  cause prioritization over direct interventions. 

Why? I don't see why longtermism wouldn't count as an important insight for cause prioritization if it was the case that thinking longtermistly didn't turn up any new intervention that we're not already known to be good, but it did change the rankings of interventions so that I changed my mind about which interventions were best. That seems to be roughly what longtermists themselves think is the situation with regard to longtermism. It's not that there is zero reason to do X-risk reduction type interventions even if LT is false, since they do benefit current people. But the case for those interventions being many times better than other things you can do for current people and animals rests on, or at least is massively strengthened by Parfit-style arguments about how there could be many happy future people. So the practical point of longtermism isn't to produce novel interventions, necessarily, but also to help us prioritize better among the interventions we already knew about. Of course, the idea of Parfit-style arguments being correct in theory is older than using it to prioritize between interventions, but so what? Why does that effect whether or not it is a good idea to use it to prioritize between interventions now? The most relevant question for what EA should fund isn't "is longtermist philosophy post -2017 simultaneously impressively original and of practical import" but "should we prioritize X-risk because of Parfit-style arguments about the number of happy people there could be in the future." If the answer to the latter question is "yes", we've agreed EAs should do what longtermists want in terms of direct work on causes, which is at least as important than how impressed we should or shouldn't be with the longtermists as researchers.* At most the latter is relevant to "should we fund more research into longtermism itself", which is important, but not as central as what first-order interventions we should fund. 
To put the point slightly differently, suppose I think the following: 

1) Based on Bostrom and Parfit-style arguments-and don't forget John Broome's case for making happy people being good-I think it's at least as influential on Will and Toby-the highest value thing to do is some form of X-risk reduction, say biorisk reduction for concreteness.

2) If it weren't for the fact that there could exist vast numbers of happy people in the far future, the benefits on the margin to current and near future people of global development work would be higher than biorisk reduction, and should be funded by EA instead, although biorisk reduction would still have significant near-term benefits, and society as a whole should have more than zero people working on it. 

Well, then I am a longtermist, pretty clearly, and it has made a difference to what I prioritize. If I am correct about 1), then it has made a good difference to what I prioritize, and if I am wrong about it, it might not have done. But it's just completely irrelevant to whether I am right to change cause prioritization based on 1) and 2) how novel 1) was if said in 2018, or what other insights LT produced as a research program.  

None of this is to say 1), or its equivalent about some other purpoted X-risk, is true. But I don't think you've said anything here that should bother someone who thinks it is. 

Year of AGI

25 years seems about right to me, but with huge uncertainty. 

I think on the racism fron Yarrow is referring to the perception that the reason Moskowtiz won't fund rationalist stuff is because either he thinks that a lot of rationalist believe Black people have lower average IQs than whites for genetic reasons, or he thinks that other people believe that and doesn't want the hassle. I think that belief genuinely is quite common among rationalists, no? Although, there are clearly rationalists who don't believe it, and most rationalists are not right-wing extremists as  far as I can tell. 

What have EA funders done that's upset you? 

Not everything being funded here even IS alignment techniques, but also, insofar as you just want general better understanding of AI as a domain through science, why wouldn't you learn useful stuff from applying techniques to current models. If the claim is that current models are too different from any possible AGI for this info to be useful, why do you think "do science" would help prepare for AGI at all? Assuming you do think that, which still seems unclear to me. 

I asked about genuine research creativity not AGI, but I don't think this conversation is going anywhere at this point. It seems obvious to me that "does stuff mathematicians say makes up the building blocks of real research" is meaningful evidence that the chance that models will do research level maths in the near future is not ultra-low, given that capabilities do increase with time. I don't think this analogous to IQ tests or the bar exam, and for other benchmarks, I would really need to see what your claiming is the equivalent of the transfer from frontier math 4 to real math that was intuitive but failed.