All of NunoSempere's Comments + Replies

There is also the interesting thing that, as far as I can tell, New Harvest, founded in 2004, basically failed, and we had to wait until the Good Food Institute came to push things along in 2016.

(as a point of comparison, New Harvest claims to have raised ~$7.5M in its frontpage (presumably during the whole of its existence), whereas the GFI spent $8.9M in 2019 alone)

Yeah, I'd appreciate that.

I liked the post. Some notes:

• In my experience, the level of disagreeableness and paranoia needed to overcome motivated reasoning is very much above the levels than a casual EA group can sustain.
• A less nice way to describe "motivated reasoning" might be "load-bearing delusions". So it's not that people can just correct their motivated reasoning to correct reasoning "just so". For instance, if someone has a need to believe that their work is useful and valuable and engage in motivated reasoning to justify this, it's not clear to me that they would at all be

until recently (with Open Phil publishing a few models for AI), longtermists have not been using numbers or models much

This seems to me to not be the case. For a very specific counterexample, AI Impacts has existed since 2015.

They model the situation, run the calculation and end up with 10^-12 and then say the probability is 10^-12.

Consider that if you're aggregating expert predictions, you might be generating probabilities too soon. Instead you could for instance interview the subject-matter experts, make the transcript available to expert forecasters, and then aggregate the probabilities of the latter. This might produce more accurate probabilities.

came back with 50% and 0.00000000001%.

I want to push back a bit against the use of 0.00000000001% in this example. In particular, I was sort of assuming that experts are kind of calibrated, and if two human experts have that sort of disagreement:

• Either this is the kind of scenario in which we're discussing how a fair coin will land, and one of the experts has seen the coin
• Or something is very, very wrong

In particular, with some light selection of experts (e.g, decent Metaculus forecasters), I think you'd almost never see this kind of scenario unless someon... (read more)

Cheers

I guess I can potentially see us changing our minds in a year's time and deciding that arithmetic mean of probabilities is better after all, or that some other method is better than both of these.

This seems very unlikely, I'll bet your $20 against my $80 that this doesn't happen.

The bias isn't that more questions resolve positively than users expect. The bias is that users expect more questions to resolve positive than actually resolve positive.

I don't get what the difference between these is.

Re: the Edit, I've added an additional paragraph to make that particular point slightly less biting. 

Also, thanks for the point around framing in terms of people's goals.

Thanks for the comment. Any thoughts on Linch's comment below?

Also, for others reading the thread, the equivalent LW post got a better answer.

prediction markets wax and wane

Sorry, I meant "prediction market platforms" wax and wane, thought the other interpretation is also correct.

do proof of work to show that you value their efforts

Yes, this makes some sense, thanks. 

People use feedback not just to determine what to improve at, but also as an overall assessment of whether they're doing a good job

Good point, thanks.

I added a paragraph to the "Models about the value of your project" to make it less biting.

Thanks. fixed.

beginning and ending the feedback with encouraging words,

So a version of this is also known as a "shit sandwich", and it's not clear to me that it is an effective pattern. In particular, it seems plausible that it only works a limited number of times before people start to notice and develop an aversion to it. I personally find it fairly irritating/annoying.

It's also not clear to me what flavor of encouragement is congruent with a situation in which e.g., getting EA jobs is particularly hard (though perhaps less so for research positions since Rethink Prio... (read more)

Thanks, changed.

Here is an update post on Metaforecast.

To resolve that prediction, yes, I would say that that interpretation is correct. 

Same.

CAUMFs might act as a fairly small layer on top of other services

If this was as easy as downloading a package over npm, it seems like an obviously good idea. But overall my impression is that this would have way too much overhead in terms of legal headache and coordination required might be pretty great.

Could we do this as a DAO

The thing this might be pointing at is that writing smarts contracts to manipulate money right now seems more convenient on a blockchain than ~anywhere else. Like, I'm sure that stripe has some API, but implementing something like a... (read more)

Thanks

My hot take: This seems like a somewhat big deal to me. It's what I would have predicted, but that's scary, given my timelines

Might be confirmation bias. But is it.

But if you already have this coalition value function, you've already solved the coordination problem and there’s no reason to actually calculate the Shapley value! If you know how much total value would be produced if everyone worked together, in realistic situations you must also know an optimal allocation of everyone’s effort. And so everyone can just do what that optimal allocation recommended.

This seems correct

A related claim is that the Shapley value is no better than any other solution to the bargaining problem. For example, instead of allocat

I think that some of your anti-expected-value beef can be addressed by considering stochastic dominance as a backup decision theory in cases where expected value fails.

For instance, maybe I think that a donation to ALLFED in expectation leads to more lives saved than a donation to a GiveWell charity. But you could point out that the expected value is undefined, because maybe the future contains infinite amount of both flourishing and suffering. Then donating to ALLFED can still be the superior option if I think that it's stochastically dominant.

There are p... (read more)

Notes on: A Sequence Against Strong Longtermism

Summary for myself. Note: Pretty stream-of-thought.

Proving too much

• The set of all possible futures is infinite which somehow breaks some important assumptions longtermists are apparently making.
  • Somehow this fails to actually bother me
• ...the methodological error of equating made up numbers with real data
  • This seems like a cheap/unjustified shot. In the world where we can calculate the expected values, it would seems fine to compare (wide, uncertain) speculative interventions with harcore GiveWell data (note that

(Edited to add Centre for the Study of Existential Risk Four Month Report June - September 2020  to the CSER sources)

Sure, but it was particularly salient to me in this case because the evaluation was so negative

In what capacity are you asking? I'd be more likely to do so if you were asking as a team member, because the organization right now looks fairly small and I would almost be evaluating individuals.

So what I specifically meant was: It's interesting that the current leadership probably thinks that CSER is valuable (e.g., valuable enough to keep working at it, rather than directing their efforts somewhere else, and presumably valuable enough to absorb EA funding and talent). This presents a tricky updating problem, where I should probably average my own impressions from my shallow review with their (probably more informed) perspective. But in the review, I didn't do that, hence the "unmitigated inside view" label. 

I like "opaqueness" for the reason that it is gradable.

This is a good question, and in hindsight, something I should have recorded. For the project as a whole, maybe two weeks to a month, but not of full-time work. I don't remember the times for each organization. 

Thanks Michael. Going through your options one by one.

1. Inform decisions about donations that are each in something like the $10-$5000 dollar range. Not an aim I had, but sure, why not.
2. Inform decisions about donations/grants that are each in something like the >$50,000 dollar range. So rather than inform those directly, inform the kind of research that you can either do or buy with money to inform that donation. $50,000 feels a little bit low for commissioning research to make a decision, though (could a $5k to $10k investment in a better version of this

Thanks for considering ALLFED. We try to respond to inquiries quickly. We have looked back, and have not be able to locate any such inquiries. We will be finalizing our 2020 report with financial details soon.

This is most likely my fault; I think I got confused between allfed.org and allfed.info

To clarify, the cost of preparation does not include the scale up in a catastrophe

For clarity:
1. Your guesstimate model: 3% to 50% mitigation of the impact of war with a 30M to 200M, a war which has a probability 0.02% to 5% per year. You also say that so far, you'v... (read more)

Thanks Michael, beautiful comment.

I think you'd have to think about the market equilibrium here. So for instance, if the price of capturing a tCO2e falls to $0.1/tonne, then more people will want to buy them, and the impact of a marginal tonne captured [1] might be lower. More generally, more people would be doing climate related projects, because the administration would be more welcoming of them.

In contrast, under $1/tonne, less people might want to buy them, and thus the marginal impact of a tonne captured might be higher. Similarly, perhaps fewer people would choose to carry out climat... (read more)

I would also want more bins than the ones I provide, i.e., not considering the total value is probably one of the parts I like less about this post. 

Makes sense. In particular, noticing that grants are all particularly legible might lead you update in the direction of a truncated distribution like you consider. So far, the LTFF seems like it has maybe moved  a bit in the direction of more legibility, but not that much.

Average yearly donation by EAs (EA survey respondents)

Suppose you give initial probability to all three normals. Then you sample an event, and its value is 1. Then you update against the green distribution, and in favor of the red and black distributions. The black distribution has a higher mean, but the red one has a higher standard deviation.

Well, because a success can be caused by a process who has a high mean, but also by a process which has a lower mean and a higher standard deviation. So for example, if you learn that someone has beaten Magnus Carlsen, it could be someone in the top 10, like Caruana, or it could be someone like Ivanchuk, who has a reputation as an "unreliable genius" and is currently number 56, but who, when he has good days, has extremely good days.

are you referring to what it seems the LTFF expected when they made the grant, what you think you would've expected at the time the grant was made, what you expect from EA/longtermist donations in general, or something else

Yes, that's tricky. The problem I have here is that different grants are in different domains and take different amounts. Ideally I'd have something like "utilons per dollar/other resources" but that's impractical. Instead, I judge a grant in its own terms: Did it achieve the purpose in the grants rationale? or something similarly valuable in case there was  a change of plan?

Thanks! To answer the questions under the first bullet point: 

• Individuals performed better than organizations, but there weren't that many organizations. 
• Individuals pursuing research directions mostly did legibly well, and the ones who didn't do legibly well seem like they had less of a well-defined plan, as one might expect. 
  • But some people with less defined directions also seem like they did well. 
  • Also note that maybe I'm rating research directions which didn't succeeded as less well defined. 
  • I don't actually have access to the

Did I imply that I thought it was bad for people to update in this way?

Reading it again, you didn't

Yes, for me updating upwards on total success on a lower percentage success rate seems intuitively fairly weird. I'm not saying it's wrong, it's that I have to stop and think about it/use my system 2. 

In particular, you have to have a prior distribution such that more valuable opportunities have a lower success rate. But then you have to have a bag of opportunities such that the worse they do, the more you get excited.

Now, I think this happens if you have a bag with "golden tickets", "sure things", and "duds".  Then not doing well would make you ... (read more)

I find the simplicity of this appealing.

If the extent of your evaluation is a quick search for public info, and you don't find much, I think the responsible conclusion is "it's unclear what happened" rather than "something went wrong". I think this holds even for projects that obviously should have public outputs if they've gone well.

So to push back against this, suppose that if you have four initial probabilities (legibly good, silently good, legibly bad, silently bad). Then you also have a ratio (legibly good + silently good) : (legibly bad + silently bad). 

Now if you learn that the proje... (read more)