All of Michael_Wiebe's Comments + Replies

Enlightenment Values in a Vulnerable World

Related, John von Neumann on x-risk:

Finally and, I believe, most importantly, prohibition of technology (invention and development, which are hardly separable from underlying scientific inquiry), is contrary to the whole ethos of the industrial age. It is irreconcilable with a major mode of intellectuality as our age understands it. It is hard to imagine such a restraint successfully imposed in our civilization. Only if those disasters that we fear had already occurred, only if humanity were already completely disillusioned about technological civilization

... (read more)
2Maxwell Tabarrok1mo
Yes, this paper is great and it was an inspiration for my piece. I found his answer here pretty unsatisfying though so hopefully I was able to expand on it well.
Should you still use the ITN framework? [Red Teaming Contest]

It sounds like you're arguing that we should estimate 'good done/additional resources' directly (via Fermi estimates), instead of indirectly using the ITN framework. But shouldn't these give the same answer?

1Karthik Tadepalli1mo
I don't think OP is opposed to multiplying them together.
Should you still use the ITN framework? [Red Teaming Contest]

And even when you can multiply the three quantities together, I feel like speaking in terms of importance, neglectedness and tractability might make you feel that there is no total ordering of intervention (“some have higher importance, some have higher tractability, whether you prefer one or the other is a matter a personal taste”)

I don't follow this. If you multiply I*T*N and get 'good done/additional resources', how is that not an ordering?

That's an ordering! It's mostly analyses like the ones of 80k Hours, which do not multiply the three together, which might let you think there is no ordering. Is there a way I can make that more precise?
It's OK not to go into AI (for students)

There seems to be a "intentions don't matter, results do" lesson that's relevant here. Intending to solve AI alignment is secondary, and doesn't mean that you're making progress on the problem.

And we don't want people saying "I'm working on AI" just for the social status, if that's not their comparative advantage and they're not actually being productive.

Yes that's exactly it! Even if a lot of people think that AI is the most important problem to work on, I would expect only a small minority to have a comparative advantage. I worry that students are setting themselves up for burnout and failure by feeling obligated to work on what's been billed as some as the most pressing/impactful cause area, and I worry that it's getting in the way of people exploring with different roles and figuring out and building out their actual comparative advantage
Person-affecting intuitions can often be money pumped

Hm, then I find necessitarianism quite strange. In practice, how do we identify people who exist regardless of our choices?

I think in ordinary cases, necessitarianism ends up looking a lot like presentism. If someone presently exists, then they exist regardless of my choices. If someone doesn't yet exist, their existence likely depends on my choices (there's probably something I could do to prevent their existence). Necessitarianism and presentism do differ in some contrived cases, though. For example, suppose I'm the last living creature on Earth, and I'm about to die. I can either leave the Earth pristine or wreck the environment. Some alien will soon be born far away and then travel to Earth. This alien's life on Earth will be much better if I leave the Earth pristine. Presentism implies that it doesn't matter whether I wreck the Earth, because the alien doesn't exist yet. Necessitarianism implies that it would be bad to wreck the Earth, because the alien will exist regardless of what I do.
An epistemic critique of longtermism

The longtermist claim is that because humans could in theory live for hundreds of millions or billions of years, and we have potential to get the risk of extinction very almost to 0, the biggest effects of our actions are almost all in how they affect the far future. Therefore, if we can find a way to predictably improve the far future this is likely to be, certainly from a utilitarian perspective, the best thing we can do.

I don't find this framing very useful. The importance-tractability-crowdedness framework gives us a sophisticated method for evaluating... (read more)

An epistemic critique of longtermism

Because of this heavy tailed distribution of interventions

Is it actually heavy-tailed? It looks like an ordered bar chart, not a histogram, so it's hard to tell what the tails are like.

What do you mean? It looks like a histogram to me Oh nevermind, I see what you mean, indeed the y axis seems to indicate the intervention, not the number of interventions. Still, wouldn't a histogram be very similar?
Announcing the Center for Space Governance

Zach and Kelly Weinersmith are writing a book on space settlement. Might be worth reaching out to them.

6Gustavs Zilgalvis1mo
Thanks for the suggestion, that sounds interesting! We'll make sure to reach out.
Fanatical EAs should support very weird projects

What do you think of the Bayesian solution, where you shrink your EV estimate towards a prior (thereby avoiding the fanatical outcomes)?

5Derek Shiller1mo
Thanks for sharing this. My (quick) reading is that the idea is to treat expected value calculations not as gospel, but as if they are experiments with estimated error intervals. These experiments should then inform, but not totally supplant, our prior. That seems sensible for givewell’s use cases, but I don’t follow the application to pascal’s mugging cases or better supported fanatical projects. The issue is that they don’t have expected value calculations that make sense to regard as experiments. Perhaps the proposal is that we should have a gut estimate and a gut confidence based on not thinking through the issues much, and another estimate based on making some guesses and plugging in the numbers, and we should reconcile those. I think this would be wrong. If anything, we should take our Bayesian prior to be our estimate after thinking through all the issues, (but perhaps before plugging in all of the exact numbers). If you’ve thought through all the issues above, I think it is appropriate to allow an extremely high expected value for fanatical projects even before trying to make a precise calculation. Or at least it is reasonable for your prior to be radically uncertain.
4Thomas Kwa1mo
There are ways to deal with Pascal's Mugger with leverage penalties [] , which IIRC deal with some problems but are not totally satisfying in extremes.
When Giving People Money Doesn't Help

The three groups have completely converged by the end of the 180 day period

I find this surprising. Why don't the treated individuals stay on a permanently higher trajectory? Do they have a social reference point, and since they're ahead of their peers, they stop trying as hard?

Person-affecting intuitions can often be money pumped

Is the difference between actualism and necessitarianism that actualism cares about both (1) people who exist as a result of our choices, and (2) people who exist regardless of our choices; whereas necessitarianism cares only about (2)?

A Critical Review of Open Philanthropy’s Bet On Criminal Justice Reform

I wonder if we can back out what assumptions the 'peace pact' approach is making about these exchange rates. They are making allocations across cause areas, so they are implicitly using an exchange rate.

A Critical Review of Open Philanthropy’s Bet On Criminal Justice Reform

I get the weak impression that worldview diversification (partially) started as an approximation to expected value, and ended up being more of a peace pact between different cause areas. This peace pact disincentivizes comparisons between giving in different cause areas, which then leads to getting their marginal values out of sync. 

Do you think there's an optimal 'exchange rate' between causes (eg. present vs future lives, animal vs human lives), and that we should just do our best to approximate it? 

Yes. To elaborate on this, I think that agents should converge on such an exchange as they become more wise and understand the world better. Separately, I think that there are exchange rates that are inconsistent with each other, and I would already consider it a win to have a setup where the exchange rates aren't inconsistent.
Kurzgesagt - The Last Human (Longtermist video)

If we don't kill ourselves in the next few centuries or millennia, almost all humans that will ever exist will live in the future.

The idea is that, after a few millenia, we'll have spread out enough to reduce extinction risks to ~0?

Even without considering that, if we stay at ~140 million births per year, in 800 years 50% of all humans will have been born in our future.
And in ~7 millennia 90% of all humans will have been born in our future.

Basically, yes. Assuming civilization survives the Singularity, existential risks are effectively zero thanks to the fact that it's almost impossible to destroy an interstellar civilization.
Results of a survey of international development professors on EA

Nice work! Sounds like movement building is very important.

Longtermist slogans that need to be retired

Do you disagree with FTX funding lead elimination instead of marginal x-risk interventions?

4Zach Stein-Perlman2mo
Not actively. I buy that doing a few projects with sharper focus and tighter feedback loops can be good for community health & epistemics. I would disagree if it took a significant fraction of funding away from interventions with a more clear path to doing an astronomical amount of good. (I almost added that it doesn't really feel like lead elimination is competing with more longtermist interventions for FTX funding, but there probably is a tradeoff in reality.)
Longtermist slogans that need to be retired

I happen to disagree that possible interventions that greatly improve the expectation of the long-term future will soon all be taken.

What do you think about MacAskill's claim that "there’s more of a rational market now, or something like an efficient market of giving — where the marginal stuff that could or could not be funded in AI safety is like, the best stuff’s been funded, and so the marginal stuff is much less clear."?

6Zach Stein-Perlman2mo
I mostly agree that obviously great stuff gets funding, but I think the "marginal stuff" is still orders of magnitude better in expectation than almost any neartermist interventions.
Longtermist slogans that need to be retired

Do you think FTX funding lead elimination is a mistake, and that they should do patient philanthropy instead?

4Jack Malde2mo
Well I’d say that funding lead elimination isn’t longtermist all other things equal. It sounds as if FTX’s motivation for funding it was for community health / PR reasons in which case it may have longtermist benefits through those channels. Whether longtermists should be patient or not is a tricky, nuanced question which I am unsure about, but I would say I’m more open to patience than most.
Critiques of EA that I want to read

Also, how are you defining "longtermist" here? You seem to be using it to mean "focused on x-risk".

Definitely mostly using it to mean focused on x-risk, but most because that seems like the largest portion / biggest focus area for the community. I interpret that Will MacAskill quote as saying that even the most hardcore longtermists care about nearterm outcomes (which seems true), not that lead reduction is supported from a longtermist perspective. I think it's definitely right that most longtermists I meet are excited about neartermist work. But I also think that the social pressures in the community currently still push toward longtermism. To be clear, I don't necessarily think this is a bad thing - it definitely could be good given how neglected longtermist issues are. But I've found the conversation around this to feel somewhat like it is missing what the critics are trying to get at, and that this dynamic is more real than people give it credit for.
Critiques of EA that I want to read

I think that these factors might be making it socially harder to be a non-longtermist who engages with the EA community, and that is an important and missing part of the ongoing discussion about EA community norms changing.

Although note that Will MacAskill supports lead elimination from a broad longtermist perspective:

Well, it’s because there’s more of a rational market now, or something like an efficient market of giving — where the marginal stuff that could or could not be funded in AI safety is like, the best stuff’s been funded, and so the marginal stu

... (read more)
Also, how are you defining "longtermist" here? You seem to be using it to mean "focused on x-risk".
Michael_Wiebe's Shortform

But again, whether non-extinction catastrophe or extinction catastrophe, if the probabilities are high enough, then both NTs and LTs will be maxing out their budgets, and will agree on policy. It's only when the probabilities are tiny that you get differences in optimal policy.

The value of x-risk reduction

Using  in  is assuming constant returns to scale. If you have , you get diminishing returns.

Messing around with some python code:

from scipy.stats import norm
import numpy as np

def risk_reduction(K,L,alpha,beta):
 print('risk:', norm.cdf(-(K**alpha)*(L**beta)))
 print('expected value:', 1/norm.cdf(-(K**alpha)*(L**beta)))
 print('risk (2x):', norm.cdf(-((2*K)**alpha)*(L**beta)))
 print('expected value (2x):', 1/norm.cdf(-((2*K)**alpha)*(L**beta)))
 print('ratio:',(1/norm.cdf(-((2... (read more)

Michael_Wiebe's Shortform

Agreed, that's another angle. NTs will only have a small difference between non-extinction-level catastrophes and extinction-level catastrophes (eg. a nuclear war where 1000 people survive vs one that kills everyone), whereas LTs will have a huge difference between NECs and ECs.

But again, whether non-extinction catastrophe or extinction catastrophe, if the probabilities are high enough, then both NTs and LTs will be maxing out their budgets, and will agree on policy. It's only when the probabilities are tiny that you get differences in optimal policy.
Michael_Wiebe's Shortform

I agree that it's a difficult problem, but I'm not sure that it's impossible.

2Charles He2mo
I don't know much about anything really, but IMO it seems really great that you are interested. There are many people with the same thoughts or interests as you. It will be interesting to see what you come up with.
Michael_Wiebe's Shortform

Yes, I think of EA as optimally allocating a budget to maximize social welfare, analogous to the constrained utility maximization problem in intermediate microeconomics. 

The worldview diversification problem is in putting everything in common units (eg. comparing human and animal lives, or comparing current and future lives). Uncertainty over these 'exchange rates' translates into uncertainty in our optimal budget allocation.

4Charles He2mo
Bruh. I just wrote out at least one good reference that EAs can’t really stick things in common units. It’s entirely possible I’m wrong, but from my personal perspective, as a general principle it seems like a good idea to identify where I’m wrong or even just describe how your instincts tell you to do something different, which can be valid. I mean for one thing, you get “fanaticism” AKA “corner solutions” for most reductive attempts to constrain max this thingy.
Michael_Wiebe's Shortform

Yes, it sounds like MacAskill's motivation is about PR and community health ("getting people out of bed in the morning"). I think it's important to note when we're funding things because of direct expected value, vs these indirect effects.

2Charles He2mo
I think what you wrote is a fair take. Just to be clear, I'm pretty sure the idea "The non-longtermist interventions are just community health and PR" is impractical and will be wobbly (a long term weakness) because: * The people leading these projects (and their large communities), who are substantial EA talent, won't at all accept the idea that they are window dressing or there to make longtermists feel good. * Many would find that a slur, and that's not healthiest to propagate from a community cohesion standpoint. * Even if the "indirect effects" model is mostly correct, it's dubious at best who gets to decide which neartermist project is a "look/feel good project" that EA should fund, and this is problematic. * Basically, as a lowly peasant, IMO, I'm OK with MacAskill, Holden deciding this, because I think there is more information about the faculty of these people and how they think and they seem pretty reasonable. * But having this perspective and decision making apparatus seems wonky. Like, will neartermist leaders just spend a lot of their time pitching and analyzing flow through effects? * $1B a year (to GiveWell) seems large for PR and community health, especially since the spend on EA human capital from those funds is lower than other cause areas To get a sense of the problems, this post here is centered entirely around [] the anomaly of EA vegan diets, which they correctly point out doesn't pass a literal cost effectiveness test. They then spend the rest of the post drawing on this to promote their alternate cause area. I think you can see how this would be problematic and self-defeating if EAs actually used this particular theory of change to fund interventions. So I think drawing the straight line here, that these interventions are just commun
Michael_Wiebe's Shortform

Does longtermism vs neartermism boil down to cases of tiny probabilities of x-risk? 

When P(x-risk) is high, then both longtermists and neartermists max out their budgets on it. We have convergence.

When P(x-risk) is low, then the expected value is low for neartermists (since they only care about the next ~few generations) and high for longtermists (since they care about all future generations). Here, longtermists will focus on x-risks, while neartermists won't.

I think for moderate to high levels of x-risk, another potential divergence is that while both longtermism and non-longtermism axiologies will lead you to believe that large scale risk prevention and mitigation is important, specific actions people take may be different. For example: * non-longtermism axiologies will all else equal be much more likely to prioritize non-existential GCRs over existential * mitigation (especially worst-case mitigation) for existential risks is comparatively more important for longtermists than for non-longtermists. Some of these divergences were covered at least as early as Parfit (1982) []. (Note: I did not reread this before making this comment). I agree that these divergences aren't very strong for traditional AGI x-risk scenarios, in those cases I think whether and how much you prioritize AGI x-risk depends almost entirely on empirical beliefs.
2Charles He2mo
I think you are very interested in cause area selection, in the sense of how these different cause areas can be "rationally allocated" in some sort of normative, analytical model that can be shared and be modified. For example, you might want such a model because you can then modify underlying parameters to create new allocations. If the model is correct and powerful, this process would illuminate what these parameters and assumptions are, laying bare and reflecting underlying insights of the world, and allowing different expression of values and principles of different people. The above analytical model is in contrast to a much more atheoretical "model", where resources are allocated by the judgement of a few people who try to choose between causes in a modest and principled way. I'm not sure your goal is possible. In short, it seems the best that can be done is for resources to be divided up, in a way bends according to principled but less legible decisions made by the senior leaders. This seems fine, or at least the best we can do. Below are some thoughts about this. The first two points sort of "span or touch on" considerations, while I think Cotra's points are the best place to start from. * The bottom half of this following comment tries to elaborate what is going on [] , as one of several points, this might be news to you: * This post by Applied Divinity Studies [] (which I suspected is being arch and slightly subversive) asking about what EAs on the forum (much less the public) are supposed to do to inform funding decisions, if at all. * (This probably isn't the point ADS wanted to make or would agree with) but my takeaway is that judgement on any cause is hard and va
AI Could Defeat All Of Us Combined

Do we know the expected cost for training an AGI? Is that within a single company's budget?

Nearly impossible to answer. This report by OpenPhil gives it a hell of an effort, but could still be wrong by orders of magnitude. Most fundamentally, the amount of compute necessary for AGI might not be related to the amount of compute used by the human brain, because we don’t know how similar our algorithmic efficiency is compared to the brain’s.

The dangers of high salaries within EA organisations

As you note, the key is being able to precisely select applicants based on altruism:

This tension also underpins a frequent argument made by policymakers that extrinsic rewards should be kept low so as to draw in agents who care sufficiently about delivering services per se. A simple conceptual framework makes precise that, in line with prevailing policy concerns, this attracts applicants who are less prosocial conditional on a given level of talent. However, since the outside option is increasing in talent, adding career benefits will draw in more talented

... (read more)
The dangers of high salaries within EA organisations

Why does your graph have financial motivation as the y-axis? Isn't financial motivation negatively correlated with altruism, by definition? In other words, financial motivation and altruism are opposite ends of a one-dimensional spectrum.

I would've put talent on the y-axis, to illustrate the tradeoff between talent and altruism.

The dangers of high salaries within EA organisations

So perhaps EA orgs can raise salaries and attract more-talented-yet-equally-commited workers. (Though this effect would depend on the level of the salary.)

AI Could Defeat All Of Us Combined

Let  be the computing power used to train the model. Is the idea that "if you could afford  to train the model, then you can also afford  for running models"? 

Because that doesn't seem obvious. What if you used 99% of your budget on training? Then you'd only be able to afford  for running models.

Or is this just an example to show that training costs >> running costs?

Yes, that's how I understood it as well. If you spend the same amount on inference as you did on training, then you get a hell of a lot of inference. I would expect he'd also argue that, because companies are willing to spend tons of money on training, we should also expect them to be willing to spend lots on inference.
2Charles He2mo
Yes, the last sentence is exactly correct. So like the terms of art here are “training” versus “inference”. I don’t have a reference or guide (because the relative size is not something that most people think about versus the absolute size of each individually) but if you google them and scroll through some papers or posts I think you will see some clear examples.
The dangers of high salaries within EA organisations


"Losing Prosociality in the Quest for Talent? Sorting, Selection, and Productivity in the Delivery of Public Services"
By Nava Ashraf, Oriana Bandiera, Edward Davenport, and Scott S. Lee


We embed a field experiment in a nationwide recruitment drive for a new health care position in Zambia to test whether career benefits attract talent at the expense of prosocial motivation. In line with common wisdom, offering career opportunities attracts less prosocial applicants. However, the trade-off exists only at low levels of talent; the marginal app

... (read more)
As you note, the key is being able to precisely select applicants based on altruism:
So perhaps EA orgs can raise salaries and attract more-talented-yet-equally-commited workers. (Though this effect would depend on the level of the salary.)
AI Could Defeat All Of Us Combined

Basically, is the computing power for training a fixed cost or a variable cost? If it's a fixed cost, then there's no further cost to using the same computing power to train models.

3Charles He2mo
I haven’t read the OP (I haven’t read a full forum post in weeks and I don’t like reading, it’s better to like, close your eyes and try coming up with the entire thing from scratch and see if it matches, using high information tags to compare with, generated with a meta model) but I think this is a referral to the usual training/inference cost differences. For example, you can run GPT-3 Davinci in a few seconds at trivial cost. But the training cost was millions of dollars and took a long time. There are further considerations. For example, finding the architecture (stacking more things in Torch, fiddling with parameters, figuring out how to implement the Key Insight , etc.) for finding the first breakthrough model is probably further expensive and hard.
AI Could Defeat All Of Us Combined

once the first human-level AI system is created, whoever created it could use the same computing power it took to create it in order to run several hundred million copies for about a year each.

How does computing power work here? Is it:

  1. We use a supercomputer to train the AI, then the supercomputer is just sitting there, so we can use it to run models. Or:
  2. We're renting a server to do the training, and then have to rent more servers to run the models.

In (2), we might use up our whole budget on the training, and then not be able to afford to run any models.

Basically, is the computing power for training a fixed cost or a variable cost? If it's a fixed cost, then there's no further cost to using the same computing power to train models.
AGI Ruin: A List of Lethalities

Great comment. Perhaps it would be helpful to explicitly split the analysis by assumptions about takeoff speed? It seems that conditional on takeoff speed, there's not much disagreement.

Potatoes: A Critical Review

This paper makes that point about linear regressions in general.

Four Concerns Regarding Longtermism

Re: discount factor,  longtermists have zero pure time preference. They still discount for exogenous extinction risk and diminishing marginal utility.


Nuclear risk research ideas: Summary & introduction

I’m very unsure how many people and how much funding the effective altruism community should be allocating to nuclear risk reduction or related research, and I think it’s plausible we should be spending either substantially more or substantially less labor and funding on this cause than we currently are (see also Aird & Aldred, 2022a).[6] And I have a similar level of uncertainty about what “intermediate goals”[7] and interventions to prioritize - or actively avoid - within the area of nuclear risk reduction (see Aird & Aldred, 2022b). Th

... (read more)
A personal take on longtermist AI governance

One possible response is about long vs short AI timelines, but that seems orthogonal to longtermism/neartermism.

A personal take on longtermist AI governance

Our AI focus area is part of our longtermism-motivated portfolio of grants,[2] and we focus on AI alignment and AI governance grantmaking that seems especially helpful from a longtermist perspective. On the governance side, I sometimes refer to this longtermism-motivated subset of work as "transformative AI governance" for relative concreteness, but a more precise concept for this subset of work is "longtermist AI governance."[3]

What work is "from a longtermist perspective" doing here? (This phrase is used 8 times in the article.) Is it: longtermists have ... (read more)

One possible response is about long vs short AI timelines, but that seems orthogonal to longtermism/neartermism.
Global health is important for the epistemic foundations of EA, even for longtermists

even if we’re coming from a position that thinks they’re not the most effective causes 

How do you interpret "most effective cause"? Is it "most effective given the current funding landscape"?

EAecon Retreat 2022: Apply Now!

The EAecon Retreat will be a ~30-person retreat for facilitating connections between EA economists of all levels. [...] We are open to applications from advanced undergraduate, master's, and early-stage Ph.D. students interested in EAeconomics who may not have yet been exposed to the area more in-depth.

So 'all levels' does not include late-stage or post-PhD economists?

8Brian Jabarian3mo
Thanks Michael for your interest and your comment: sure thing it does, fixed.
4Brian Jabarian3mo
Thank you !
Load More