Effective Altruism Forum
EA Forum

All of CarlShulman's Comments + Replies

Nuclear war tail risk has been exaggerated?

Rapid fire:

Nearterm extinction risk from AI is wildly closer to total AI x-risk than the nuclear analog
My guess is that nuclear war interventions powerful enough to be world-beating for future generations would look tremendous in averting current human deaths, and most of the WTP should come from that if one has a lot of WTP related to each of those worldviews
Re suspicious convergence, what do you want to argue with here? I've favored allocation on VOI and low-hanging fruit on nuclear risk not leveraging AI related things in the past less than 1% of

... (read more)

Vasco Grilo

2mo

Thanks for following up! Sorry for the lack of clarity. Some thoughts: * The 15.3 M$ grantmakers aligned with effective altruism have influenced aiming to decrease nuclear risk seem mostly optimised to decrease the nearterm damage caused by nuclear war (especially the spending on nuclear winter), not the more longterm existential risk linked to permanent global totalitarianism. * As far as I know, there has been little research on how a minor AI catastrophe would influence AI existential risk (although wars over Taiwan have been wargamed). Looking into this seems more relevant than investigating how a non-AI catastrophe would influence AI risk. * The risk from permanent global totalitarianism is still poorly understood, so research on this and how to mitigate it seems more valuable than efforts focussing explicitly on nuclear war. There might well be interventions to increase democracy levels in China which are more effective to decrease that risk than interventions aimed at ensuring that China does not become the sole global hegemon after a nuclear war. * I guess most of the risk from permanent global totalitarianism does not involve any major catastrophes. As a data point, the Metaculus' community predicts an AI dystopia is 5 (= 0.19/0.037) times as likely as a paperclipalypse by 2050. More broadly, which pieces would you recommend reading on this topic? I am not aware of substantial blogposts, although I have seen the concern raised many times.

Nuclear war tail risk has been exaggerated?

CarlShulman2mo58

I agree that people should not focus on nuclear risk as a direct extinction risk (and have long argued this), see Toby's nuke extinction estimates as too high, and would assess measures to reduce damage from nuclear winter to developing neutral countries mainly in GiveWell-style or ordinary CBA terms, while considerations about future generations would favor focus on AI, and to a lesser extent bio.

However, I do think this wrongly downplays the effects on our civilization beyond casualties and local damage of a nuclear war that wrecks the current nucl... (read more)

Vasco Grilo

2mo

Thanks for sharing your thoughts, Carl! Thanks for mentioning these points. Would you also rely on ordinary CBAs to assess interventions to decrease the direct damage of nuclear war? I think this would still make sense. At the same time, the nearterm extinction risk from AI also misses most of the existential risk from AI? I guess you are implying that the ratio between nearterm extinction risk and total existential risk is lower for nuclear war than for AI. Related to your point above, I say that: Regarding: Note I mention right after this that: You say that: I agree these are relevant considerations. On the other hand: * The US may want to attack China in order not to relenquish its position as global hegemon. * I feel like there has been little research on questions like: * How much it would matter if powerful AI was developped in the West instead of China (or, more broadly, in a democracy instead of autocracy). * The likelihood of lock-in. On the last point, your piece is a great contribution, but you say: However, the likelihood of lock-in is crucial to assess the strength of your points. I would not be surprised if the chance of an AI lock-in due to a nuclear war was less than 10^-8 this century. In terms of nuclear war indirectly causing extinction: In contrast, if powerful AI caused extinction, control over the future would arguably permanently be lost. Agreed. Is there any evidence for this? Makes sense. If GiveWell's top charities are not a cost-effective way of improving the longterm future, then decreasing starvation in low income countries in a nuclear winter may be cost-effective in terms of saving lives, but has semingly negligible impact on the longterm future too. Such countries just have too little influence on transformative technologies.

Open Phil Should Allocate Most Neartermist Funding to Animal Welfare

CarlShulman4mo13

Thank you for the comment Bob.

I agree that I also am disagreeing on the object-level, as Michael made clear with his comments (I do not think I am talking about a tiny chance, although I do not think the RP discussions characterized my views as I would), and some other methodological issues besides two-envelopes (related to the object-level ones). E.g. I would not want to treat a highly networked AI mind (with billions of bodies and computation directing them in a unified way, on the scale of humanity) as a millionth or a billionth of the welfare of ... (read more)

MichaelStJules

4mo

(Speaking for myself only.) FWIW, I think something like conscious subsystems (in huge numbers in one neural network) is more plausible by design in future AI. It just seems unlikely in animals because all of the apparent subjective value seems to happen at roughly the highest level where everything is integrated in an animal brain. Felt desire seems to (largely) be motivational salience, a top-down/voluntary attention control function driven by high-level interpretations of stimuli (e.g. objects, social situations), so relatively late in processing. Similarly, hedonic states depend on high-level interpretations, too. Or, according to Attention Schema Theory, attention models evolved for the voluntary control of attention. It's not clear what the value would be for an attention model at lower levels of organization before integration. And evolution will select against realizing functions unnecessarily if they have additional costs, so we should provide a positive argument for the necessary functions being realized earlier or multiple times in parallel that overcomes or doesn't incur such additional costs. So, it's not that integration necessarily reduces value; it's that, in animals, all the morally valuable stuff happens after most of the integration, and apparently only once or in small number. In artificial systems, the morally valuable stuff could instead be implemented separately by design at multiple levels. EDIT: I think there's still crux about whether realizing the same function the same number of times but "to a greater degree" makes it more morally valuable. I think there are some ways of "to a greater degree" that don't matter, and some that could. If it's only sort of (vaguely) true that a system is realizing a certain function, or it realizes some but not all of the functions possibly necessary for some type of welfare in humans, then we might discount it for only meeting lower precisifications of the vague standards. But adding more neurons ju

What are the biggest conceivable wins for animal welfare by 2025?

Answer by CarlShulmanDec 11, 202310

Lots of progress on AI, alignment, and governance. This sets up a position where it is likely that a few years later there's an AI capabilities explosion and among other things:

Mean human wealth skyrockets, while AI+robots make cultured meat and substitutes, as well as high welfare systems (and reengineering biology) cheap relative to consumers' wealth; human use of superintelligent AI advisors leads to global bans on farming with miserable animals and/or all farming
Perfect neuroscientific and psychological knowledge of humans and animals, combined w

CarlShulman4mo9

Not much new on that front besides continuing to back the donor lottery in recent years, for the same sorts of reasons as in the link, and focusing on research and advising rather than sourcing grants.

I'm interviewing Carl Shulman — what should I ask him?

CarlShulman4mo9

A bit, but more on the willingness of AI experts and some companies to sign the CAIS letter and lend their voices to the view 'we should go forward very fast with AI, but keep an eye out for better evidence of danger and have the ability to control things later.'

My model has always been that the public is technophobic, but that 'this will be constrained like peaceful nuclear power or GMO crops' isn't enough to prevent a technology that enables DSA and OOMs (and nuclear power and GMO crops exist, if AGI exists somewhere that place outgrows the rest of the w... (read more)

Open Phil Should Allocate Most Neartermist Funding to Animal Welfare

CarlShulman4mo6

I don't want to convey that there was no discussion, thus my linking the discussion and saying I found it inadequate and largely missing the point from my perspective. I made an edit for clarity, but would accept suggestions for another.

MichaelStJules

4mo

Your edit looks good to me. Thanks!

I'm interviewing Carl Shulman — what should I ask him?

CarlShulman4mo2

I have never calculated moral weights for Open Philanthropy, and as far as I know no one has claimed that. The comment you are presumably responding to began by saying I couldn't speak for Open Philanthropy on that topic, and I wasn't.

Open Phil Should Allocate Most Neartermist Funding to Animal Welfare

CarlShulman4mo4

Thanks, I was referring to this as well, but should have had a second link for it as the Rethink page on neuron counts didn't link to the other post. I think that page is a better link than the RP page I linked, so I'll add it in my comment.

MichaelStJules

4mo

(Again, not speaking on behalf of Rethink Priorities, and I don't work there anymore.) (Btw, the quote formatting in your original comment got messed up with your edit.) I think the claims I quoted are still basically false, though? Do Brains Contain Many Conscious Subsystems? If So, Should We Act Differently? explicitly considered a conscious subsystems version of this thought experiment, focusing on the more human-favouring side when you normalize by small systems like insect brains, which is the non-obvious side often neglected. There's a case that conscious subsystems could dominate expected welfare ranges even without intertheoretic comparisons (but also possibly with), so I think we were focusing on one of strongest and most important arguments for humans potentially mattering more, assuming hedonism and expectational total utilitarianism. Maximizing expected choiceworthiness with intertheoretic comparisons is controversial and only one of multiple competing approaches to moral uncertainty. I'm personally very skeptical of it because of the arbitrariness of intertheoretic comparisons and its fanaticism (including chasing infinities, and lexically higher and higher infinities). Open Phil also already avoids making intertheoretic comparisons, but was more sympathetic to normalizing by humans if it were going to.

Open Phil Should Allocate Most Neartermist Funding to Animal Welfare

CarlShulman4mo15

I'm not planning on continuing a long thread here, I mostly wanted to help address the questions about my previous comment, so I'll be moving on after this. But I will say two things regarding the above. First, this effect (computational scale) is smaller for chickens but progressively enormous for e.g. shrimp or lobster or flies. Second, this is a huge move and one really needs to wrestle with intertheoretic comparisons to justify it:

I guess we should combine them using a weighted geometric mean, not the weighted mean as I did above.

Suppose we... (read more)

Vasco Grilo

4mo

Fair, as this is outside of the scope of the original post. I noticed you did not comment on RP's neuron counts post. I think it would be valuable if you commented there about the concerns you expressed here, or did you already express them elsewhere in another post of RP's moral weight project sequence? I agree that is the case if one combines the 2 wildly different estimates for the welfare range (e.g. one based on the number of neurons, and another corresponding to RP's median welfare ranges) with a weighted mean. However, as I commented above, using the geometric mean would cancel the effect. Is this a good analogy? Maybe not: * Broadly speaking, giving the same weight to multiple estimates only makes sense if there is wide uncertainty with respect to which one is more reliable. In the example above, it would make sense to give negligible weight to all metrics except for the aggregate mass. In contrast, there is arguably wide uncertainty with respect to what are the best models to measure welfare ranges, and therefore distributing weights evenly is more appropriate. * One particular model on which we can put lots of weight on is that mass is straightforwardly additive (at least at the macro scale). So we can say the mass of all humans equals the number of humans times the mass per human, and then just estimate this for a typical human. In contrast, it is arguably unclear whether one can obtain the welfare range of an animal by e.g. just adding up the welfare range of its individual neurons.

Open Phil Should Allocate Most Neartermist Funding to Animal Welfare

CarlShulman4mo156

I can't speak for Open Philanthropy, but I can explain why I personally was unmoved by the Rethink report (and think its estimates hugely overstate the case for focusing on tiny animals, although I think the corrected version of that case still has a lot to be said for it).

Luke says in the post you linked that the numbers in the graphic are not usable as expected moral weights, since ratios of expectations are not the same as expectations of ratios.

However, I say "naively" because this doesn't actually work, due to two-envelope effects...whenev

... (read more)

MichaelDickens

1mo

It seems to me that the naive way to handle the two envelopes problem (and I've never heard of a way better than the naive way) is to diversify your donations across two possible solutions to the two envelopes problem: * donate half your (neartermist) money on the assumption that you should use ratios to fixed human value * donate half your money on the assumption that you should fix the opposite way (eg fruit flies have fixed value) Which would suggest donating half to animal welfare and probably half to global poverty. (If you let moral weights be linear with neuron count, I think that would still favor animal welfare, but you could get global poverty outweighing animal welfare if moral weight grows super-linearly with neuron count.) Plausibly there are other neartermist worldviews you might include that don't relate to the two envelopes problem, e.g. a "only give to the most robust interventions" worldview might favor GiveDirectly. So I could see an allocation of less than 50% to animal welfare.

Bob Fischer4mo31

Thanks for your discussion of the Moral Weight Project's methodology, Carl. (And to everyone else for the useful back-and-forth!) We have some thoughts about this important issue and we're keen to write more about it. Perhaps 2024 will provide the opportunity!

For now, we'll just make one brief point, which is that it’s important to separate two questions. The first concerns the relevance of the two envelopes problem to the Moral Weight Project. The second concerns alternative ways of generating moral weights. We considered the two envelopes problem at some... (read more)

MichaelStJules4mo18

It is not unthinkably improbable that an elephant brain where reinforcement from a positive or negative stimulus adjust millions of times as many neural computations could be seen as vastly more morally important than a fruit fly, just as one might think that a fruit fly is much more important than a thermostat (which some suggest is conscious and possesses preferences). Since on some major functional aspects of mind there are differences of millions of times, that suggests a mean expected value orders of magnitude higher for the elephant if you put a bit

... (read more)

MichaelStJules4mo13

(I'm not at Rethink Priorities anymore, and I'm not speaking on their behalf.)

Rethink's work, as I read it, did not address that central issue, that you get wildly different results from assuming the moral value of a fruit fly is fixed and reporting possible ratios to elephant welfare as opposed to doing it the other way around.
(...)

Rethink's discussion of this almost completely sidestepped the issue in my view.

RP did in fact respond to some versions of these arguments, in the piece Do Brains Contain Many Conscious Subsystems? If So, Should We Act Di... (read more)

RedStateBlueState4mo18

This consideration is something I had never thought of before and blew my mind. Thank you for sharing.

Hopefully I can summarize it (assuming I interpreted it correctly) in a different way that might help people who were as befuddled as I was.

The point is that, when you have probabilistic weight to two different theories of sentience being true, you have to assign units to sentience in these different theories in order to compare them.

Say you have two theories of sentience that are similarly probable, one dependent on intelligence and one depend... (read more)

Vasco Grilo4mo12

Thanks for elaborating, Carl!

Luke says in the post you linked that the numbers in the graphic are not usable as expected moral weights, since ratios of expectations are not the same as expectations of ratios.

Let me try to restate your point, and suggest why one may disagree. If one puts weight w on the welfare range (WR) of humans relative to that of chickens being N, and 1 - w on it being n, the expected welfare range of:

Humans relative to that of chickens is E("WR of humans"/"WR of chickens") = w*N + (1 - w)*n.
Chickens relative to that of humans is E("WR

CarlShulman5mo28

One can value research and find it informative or worth doing without being convinced of every view of a given researcher or team. Open Philanthropy also sponsored a contest to surface novel considerations that could affect its views on AI timelines and risk. The winners mostly present conclusions or considerations on which AI would be a lower priority, but that doesn't imply that the judges or the institution changed their views very much in that direction.

At large scale, Information can be valuable enough to buy even if it only modestly adjusts pro... (read more)

Will Aldred5mo28

Thank you for engaging. I don’t disagree with what you’ve written; I think you have interpreted me as implying something stronger than what I intended, and so I’ll now attempt to add some colour.

That Emily and other relevant people at OP have not fully adopted Rethink’s moral weights does not puzzle me. As you say, to expect that is to apply an unreasonably high funding bar. I am, however, puzzled that Emily and co. appear to have not updated at all towards Rethink’s numbers. At least, that’s the way I read:

We don’t use Rethink’s moral weights.
Our cur

CarlShulman6mo5

there not being enough practically accessible matter available (even if we only ever need a finite amount), and

This is what I was thinking about. If I need a supply of matter set aside in advance to be able to record/receive an answer, no finite supply suffices. Only an infinite brain/tape, or infinite pile of tape making resources, would suffice.

If the resources are created on demand ex nihilo, and in such a way that the expansion processes can't be just 'left on' you could try to jury rig around it.

MichaelStJules

6mo

The resources wouldn't necessarily need to be created on demand ex nihilo either (although that would suffice), but either way, we're forced into extremely remote possibilities — denying our current best understanding of physics — and perhaps less likely than infinite accessible resources (or other relevant infinities). That should be enough to say it's less conservative than actual infinities and make your point for this particular money pump, but it again doesn't necessarily depend on actual infinities. However, some people actually assign 0 probability to infinity (I think they're wrong to do so), and some of them may be willing to grant this possibility instead. For them, it would actually be more conservative. The resources could just already exist by assumption in large enough quantities by outcome in the prospect (at least with nonzero probability for arbitrarily large finite quantities). For example, the prospect could be partially about how much information we can represent to ourselves (or recognize). We could be uncertain about how much matter would be accessible and how much we could do with it. So, we can have uncertainty about this and may not be able to put an absolute hard upper bound on it with certainty, even if we could with near-certainty, given our understanding of physics and the universe, and our confidence in them. And this could still be the case conditional on no infinities. So, we could consider prospects with extremely low probability heavy tails for how much we could represent to ourselves, which would have the important features of St Petersburg prospects for the money pump argument. It’s also something we'd care about naturally, because larger possible representations would tend to coincide with much more possible value. St Petersburg prospects already depend on extremely remote possibilities to be compelling, so if you object to extremely low probabilities or instead assign 0 probability to them (deny the hypothetical), then you can

Arguments for utilitarianism are impossibility arguments under unbounded prospects

CarlShulman6mo2

I personally think unbounded utility functions don't work, I'm not claiming otherwise here, the comment above is about the thought experiment.

Arguments for utilitarianism are impossibility arguments under unbounded prospects

CarlShulman6mo5

Now, there’s an honest and accurate genie — or God or whoever’s simulating our world or an AI with extremely advanced predictive capabilities — that offers to tell you exactly how $A$ will turn out.^[9] Talking to them and finding out won’t affect $A$ or its utility, they’ll just tell you what you’ll get.

This seems impossible, for the possibilities that account for ~all the expected utility (without which it's finite)? You can't fit enough bits in a human brain or lifetime (or all accessible galaxies, or whatever). Your brain would... (read more)

MichaelStJules

6mo

My post also covers two impossibility theorems that don't depend on anyone having arbitrary precision or unbounded or infinite representations of anything:[1] 1. Stochastic Dominance, Anteriority and Impartiality are jointly inconsistent. 2. Stochastic Dominance, Separability and Compensation (Impartiality) are jointly inconsistent. The proofs are also of course finite, and the prospects used have finite representations, even though they represent infinitely many possible outcomes and unbounded populations. 1. ^ The actual outcome would be an unbounded (across outcomes) representation of itself, but that doesn't undermine the argument.

MichaelStJules

6mo

It wouldn't have to definitely be infinite, but I'd guess it would have to be expandable to arbitrarily large finite sizes, with the size depending on the outcome to represent, which I think is also very unrealistic. I discuss this briefly in my Responses section. Maybe not impossible if we're dealing with arbitrarily long lives, because we could keep expanding over time, although there could be other practical physical limits on this that would make this impossible, maybe requiring so much density that it would collapse into a black hole?

Nuclear war is unlikely to cause human extinction

CarlShulman8mo4

Alone and directly (not as a contributing factor to something else later), enough below 0.1% that I evaluate nuclear interventions based mainly on their casualties and disruption, not extinction. I would (and have) support them in the same kind of metric as GiveWell, not in extinction risk.

In the event of all-out WMD war (including with rogue AGI as belligerent) that leads to extinction nukes could be a contributing factor combined with bioweapons and AI (strategic WMD war raises the likelihoods of multiple WMDs being used together).

Vasco Grilo

8mo

Thanks for the reply! I agree the direct/nearterm extinction risk posed by nuclear war alone would be quite low (maybe of the order of 10^-6 in the next 100 years), but I wonder whether it could still meaningfully decrease the value of the longterm future if thousands of nukes are detonated. Models S and E of Denkenberger 2022 consider a full scale nuclear war would decrease such value by 24 % and 7 %. I think these are too high, but guess a nuclear war involving thousands of nukes might still increase indirect/longterm extinction risk to a significant extent. So I would say they are not directly comparable to GiveWell's top charities. Maybe you think they are comparable given the low likelihood of civilisation collapse, and the flow-through effects of saving lives (which might include decreasing longterm extinction risk)?

An appeal to people who are smarter than me: please help me clarify my thinking about AI

CarlShulman8mo30

>It's plausible humans will go extinct from AI. It's also plausible humans will go extinct from supervolcanoes.

Our primitive and nontechnological ancestors survived tens of millions of years of supervolcano eruptions (not to mention mass extinctions from asteroid/comet impacts) and our civilization's ability to withstand them is unprecedentedly high and rapidly increasing. That's not plausible, it's enormously remote, well under 1/10,000 this century.

NickLaing

8mo

I agree with what I think you intend to say, but in my mind plausible = any chance at all.

What would it look like for AIS to no longer be neglected?

CarlShulman10mo4

I think there are whole categories of activity that are not being tried by the broader world, but that people focused on the problem attend to, with big impacts in both bio and AI. It has its own diminishing returns curve.

What would it look like for AIS to no longer be neglected?

Answer by CarlShulmanJun 16, 202331

The thing to see is if the media attention translates into action with more than a few hundred people working on the problem as such rather than getting distracted, and government prioritizing it in conflict with competing goals (like racing to the precipice). One might have thought Covid-19 meant that GCBR pandemics would stop being neglected, but that doesn't seem right. The Biden administration has asked for Congressional approval of a pretty good pandemic prevention bill (very similar to what EAs have suggested) but it has been ... (read more)

Vasco Grilo

10mo

Hi Carl, Do you have any thoughts on how the expected impact of the few hundred people working most directly on AGI safety compares with that of the rest of the world (on mitigating the risks from advanced AI)? I suppose a random person from the few hundred people will have much greater (positive/negative) impact than a random person, but it is not obvious to me that we can round the (positive/negative) impact of the rest of the world (on mitigating the risks from advanced AI) to zero, and increased awareness of the rest of the world will tend to increase its expected impact (for better or worse). In terms of bio risk, even if covid did not result in more people working towards fighting existential pandemics, it may still have increased the number of people working on pandemics, which arguably has positive flow-through effects to mitigate the existential ones. In addition, covid may have increased the amount of resources that would be directed towards fighting a pandemic if one happens. It is unclear to me how large these effects are relative to a given increase in the number of people working most directly on reducing the risk from existential pandemics.

harfe10mo11

I feel like this does not really address the question?

A possible answer to Rockwell's question might be "If we have 15000 scientists working full-time on AIS, then I consider AIS to no longer be neglected" (this is hypothetical, I do not endorse it. And its also not as contextualized as Rockwell would want it).

But maybe I am interpreting the question too literally and you are making a reasonable guess what Rockwell wants to hear.

EAG talks are underrated IMO

CarlShulman1y9

I actually do every so often go over the talks from the past several EAGs on Youtube and find it does better. Some important additional benefits are turning on speedup and subtitles, being able to skip forward or bail more easily if the talk turns out bad, and not being blocked from watching two good simultaneous talks.

In contrast, a lot of people really love in-person meetings compared to online video or phone.

No, the EMH does not imply that markets have long AGI timelines

CarlShulman1y5

As Tom says, sorry if I wasn't clear.

Jakob

Yes, in isolation I see how that seems to clash with what Carl is saying. But that’s after I’ve granted the limited definition of TAI (x-risk or explosive, shared growth) from the former post. When you allow for scenarios with powerful AI where savings still matter, the picture changes (and I think that’s a more accurate description of the real world). I see that I could’ve been more clear that this post was a case of “even if blindly accepting the (somewhat unrealistic) assumptions of another post, their conclusions don’t follow”, and not an attempt at describing reality as accurately as possible

No, the EMH does not imply that markets have long AGI timelines

CarlShulman1y47

I disagree with the idea that short AI timelines are not investable (although I agree interest rates are a bad and lagging indicator vs AI stocks). People foreseeing increased expectations of AI sales as a result of scaling laws, shortish AI timelines, and the eventual magnitude of success have already made a lot of money investing in Nvidia, DeepMind and OpenAI. Incremental progress increases those expectations, and they can increase even in worlds where AGI winds up killing or expropriating all investors so long as there is some expectation of enough inv... (read more)

Jakob1y11

Carl, I agree with everything you're saying, so I'm a bit confused about why you think you disagree with this post.

This post is a response to the very specific case made in an earlier forum post, where they use a limited scenario to define transformative AI, and then argue that we should see interest rates rising if if traders believe that scenario to be near.

I argue that we can't use interest rates to judge if said, specific scenario is near or not. That doesn't mean there are no ways to bet on AI (in a broader sense). Yes, when tech firms are tradi... (read more)

Where I'm at with AI risk: convinced of danger but not (yet) of doom

CarlShulman1y22

If you haven't read this piece by Ajeya Cotra, Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover I would highly recommend it. Some of the post on AI alignment here (aimed at a general audience) might also be helpful.

Amber Dawn

Thanks, I'll check out the Cotra post. I've have skimmed some of the Cold Takes posts and not found where he addresses the specific confusions I have above.

Why AI is Harder Than We Think - Melanie Mitchell

CarlShulman1y7

This tweet seems like vague backtracking on the long timelines.

We're Lincoln Quirk & Ben Kuhn from Wave, AMA!

CarlShulman1y4

Well Musk was the richest, who notably pulled out and then the money seems mostly not to have manifested. I haven't seen a public breakdown of commitments those sorts of statements were based on.

MarkusAnderljung

Semafor reporting confirms your view. They say Musk promised $1bn and gave $100mn before pulling out.

AGI in sight: our look at the game board

CarlShulman1y5

The kind of examples people used to use to motivate frame problem stories in the days of GOFAI in the 20th century are routinely solved by AI systems today.

Lixiang

Interesting, well maybe I'm off base then.

There can be highly neglected solutions to less-neglected problems

CarlShulman1y5

I was going from this: "The DICE baseline emissions scenario results in 83 million cumulative excess deaths by 2100 in the central estimate. Seventy-four million of these deaths can be averted by pursuing the DICE-EMR optimal emissions path." I didn't get into deaths vs DALYs (excess deaths among those with less life left to live), chances of scenarios, etc, and gave 'on the order of' for slack.

"But I don't see why we're talking about scale. Are you defining neglectedness as a ratio of <people potentially killed in worst case>/<dollars spent>?"... (read more)

Arepo

I'm happy to leave it there, but to clarify I'm not claiming 'no difference in the type of work they do', but rather 'no a priori reason to write one group off as "not concerned with safety"'.

There can be highly neglected solutions to less-neglected problems

CarlShulman1y29

In this 2022 ML survey the median credence on extinction-level catastrophe from AI is 5%, with 48% of respondents giving 10%. Some generalist forecaster platforms put numbers significantly lower, some forecasting teams or researchers with excellent forecasting records and more knowledge of the area put more (with I think the tendency being for more information to yield higher forecasts, and my own expectation). This scale looks like hundreds of millions of deaths or equivalent this century to me, although certainly many disagree. The argu... (read more)

Arepo1y12

order 100M deaths.

I'm not sure this is super relevant to our core disagreement (if we have one), but how are you counting this? Glancing at that article, it looks like a pessimistic take on climate change's harm puts excess deaths at around 10m per year, and such damage would persist much more than 10 years.

But I don't see why we're talking about scale. Are you defining neglectedness as a ratio of <people potentially killed in worst case>/<dollars spent>?

How exactly could that be true?

Because coders who don't work explicitly on AI alignment sti... (read more)

There can be highly neglected solutions to less-neglected problems

CarlShulman1y15

$200B includes a lot of aid aimed at other political goals more than humanitarian impact, , with most of a billion people living at less than $700/yr, while the global economy is over $100,000B and cash transfer programs in rich countries are many trillions of dollars. That's the neglectedness that bumps of global aid interventions relative to local rich country help to the local relative poor.

You can get fairly arbitrarily bad cost-effectiveness in any area by taking money and wasting on it things that generate less value than the money. E.g. spending 99.9% on digging holes and filling them in, and 0.1% on GiveDirectly. But just handing over the money to the poor is a relevant attainable baseline.

Arepo1y11

Calling an area neglected because a lot of money is spent badly sounds like a highly subjective evaluation that's hard to turn into a useful principle. Sure, $200B annually is a small proportion of the global economy, but so is almost any cause area you can describe. From a quick search, the World Bank explicitly spends slightly more than a tenth of that on climate change, one of the classically 'non-neglected' evaluands of EA. It's hard to know how to compare these figures, since they obviously omit a huge number of other projects, but I doubt the WB cons... (read more)

There can be highly neglected solutions to less-neglected problems

CarlShulman1y49

Helping the global poor is neglected, and that accounts for most bednet outperformance. GiveDirectly, just giving cash, is thought by GiveWell/GHW to be something like 100x better on direct welfare than rich country consumption (although indirect effects reduce that gap), vs 1000x+ for bednets. So most of the log gains come from doing stuff with the global poor at all. Then bednets have a lot of their gains as positive externalities (protecting one person also protects others around them), and you're left with a little bit of 'being more confident about be... (read more)

Arepo

Official development assistance is nearly $200billion annually. I think if that's going to be called 'neglected', the term needs some serious refinement. People have compared various development interventions like antiretroviral drugs for HIV, which have the same positive externalities and (at least according to a presentation Toby Ord gave a few years ago), still something like 100fold difference in expected outcomes from AMF.

Why People Use Burner Accounts: A Commentary on Blacklists, the EA "Inner Circle", and Funding

CarlShulman1y59

Here's an example of a past case where a troll (who also trolled other online communities) made up multiple sock-puppet accounts, and assorted lies about sources for various arguments trashing AI safety, e.g. claiming to have been at events they were not and heard bad things, inventing nonexistent experts who supposedly rejected various claims, creating fake testimonials of badness, smearing people who discovered the deception, etc.

What's the social/historical story of EA's longtermist turn?

Answer by CarlShulmanFeb 06, 202331

https://www.openphilanthropy.org/research/three-key-issues-ive-changed-my-mind-about/

RobBensinger1y11

Came here to cite the same thing! :)

Note that Dustin Moskovitz says he's not a longtermist, and "Holden isn't even much of a longtermist."

So my intuition is that the two main important updates EA has undergone are "it's not that implausible that par-human AI is coming in the next couple of decades" and "the world is in fact dropping the ball on this quite badly, in the sense that maybe alignment isn't super hard, but to a first approximation no one in the field has checked."

(Which is both an effect and a cause of updates like "maybe we can figure stu... (read more)

AGI and the EMH: markets are not expecting aligned or unaligned AI in the next 30 years

CarlShulman1y31

But the stocks are the more profitable and capital-efficient investment, so that's where you see effects first on market prices (if much at all) for a given number of traders buying the investment thesis. That's the main investment on this basis I see short timelines believers making (including me), and has in fact yielded a lot of excess returns since EAs started to identify it in the 2010s.
I don't think anyone here is arguing against the no-trade theorem, and that's not an argument that prices will never be swayed by anything, but that you ca

CarlShulman1y37

If investors with $1T thought AGI soon, and therefore tried to buy up a portfolio of semiconductor, cloud, and AI companies (a much more profitable and capital-efficient strategy than betting on real interest rates) they could only a buy a small fraction of those industries at current prices. There is a larger pool of investors who would sell at much higher than current prices, balancing that minority.

Yes, it's weighted by capital and views on asset prices, but still a small portion of the relevant capital trying to trade (with risk and years in advance) o... (read more)

basil.halperin

1. We would welcome engagement from you regarding our argument that stock prices are not useful for forecasting timelines (the sign is ambiguous and effect noisy). 2. You offer what is effectively a full general argument against market prices ever being swayed by anything -- a bit more on this point here. Price changes do not need to be driven by volume! (cf the no-trade theorem, for the conceptual idea) 3. I'm not sure if this is exactly your point about prediction markets (or if you really want to talk about total capital, on which see again #2), but: Sovereign debt markets are orders of magnitude larger than PredictIt or other political prediction markets. These are not markets where individual traders are capped to $600 max positions and shorting is limited (or whatever the precise regulations are)! Finding easy trades in these markets is ...not easy.

Misleading phrase in a GiveWell Youtube ad

CarlShulman1y4

Same here.

What grants has Carl Shulman's discretionary fund made?

CarlShulman1y8

They still have not published. You can email Jan Brauner and Fabienne Sandkuehler for it.

How Many Lives Does X-Risk Work Save From Nonexistence On Average?

CarlShulman1y5

Expected lives saved and taken are both infinite, yes.

How Many Lives Does X-Risk Work Save From Nonexistence On Average?

CarlShulman1y14

That there are particular arguments for decisions like bednets or eating sandwiches to have expected impacts that scale with the scope of the universes or galactic civilizations. E.g. the more stars you think civilization will be able to colonize, or the more computation that will be harvested, the greater your estimate of the number of sims in situations like ours (who will act the same as we do, so that on plausible decision theories we should think of ourselves as setting policy at least for the psychologically identical ones). So if you update to... (read more)

Jordan Arel

Ah yes! I think I see what you mean. I hope to research topics related to this in the near future, including in-depth research on anthropics, as well as on what likely/desirable end-states of the universe are (including that we may already be in an end-state simulation) and what that implies for our actions. I think this could be a 3rd reason for acting to create a high amount of well-being for those close to you in proximity, including yourself.

How Many Lives Does X-Risk Work Save From Nonexistence On Average?

CarlShulman1y11

This sort of estimate is in general off by many orders of magnitude for thinking about the ratio of impact between different interventions when it only considers paths to very large numbers for the intervention under consideration, and not to reference interventions being compared against. For example, the expected number of lives saved from giving a bednet is infinite. Connecting to size-of-the-accessible-universe estimates, perhaps there are many simulations of situations like ours at an astronomical scale, and so our decisions will be replicated a... (read more)

Fermi–Dirac Distribution

I want to point out something that I find confusing. This can't be true unless your credence that you're killing an infinite number of lives by buying a bednet is exactly zero, right? Otherwise -- if your credence is, say, 10−1010101010-- then the expected number of lives saved is undefined. Am I thinking about this correctly?

Linch

I agree with the rest of your comment, but I'm a bit confused about this phrasing.

Jordan Arel

Hey Carl! Thanks for your comment. I am not sure I understand. Are you arguing something like “comparing x-risk interventions to other inventions such as bed nets is invalid because the universe may be infinite, or there may be a lot of simulations, or some other anthropic reason may make other interventions more valuable”?

Getting on a different train: can Effective Altruism avoid collapsing into absurdity?

CarlShulman2y27

The implicit utility function in Kelly (log of bankroll) amounts to rejecting additive aggregation/utilitarianism. That would be saying that doubling goodness from 100 to 200 would be of the same decision value as doubling from 100 billion to 200 billion, even though in the latter case the benefit conferred is a billion times greater.

It also absurdly says that loss goes to infinity as you go to zero. So it will reject any finite benefit of any kind to prevent even an infinitesimal chance of going to zero. If you say that the world ending has infinit... (read more)

DirectedEvolution

That's a helpful correction/clarification, thank you! I suppose this is why it's important to be cautious about overapplying a particular utilitarian calculation - you (or in this case, I) might be wrong in how you're going about it, even though the right ultimate conclusion is justified on the basis of a correct utilitarian calculus.

NASA will re-direct an asteroid tonight as a test for planetary defence (link-post)

CarlShulman2y14

This is much more of a problem (and an overwhelming one) for risks/opportunities that are microscopic compared to others. Baseline asteroid/comet risk is more like 1 in a billion. Much less opportunity for that with 1% or 10% risks.

Critique of MacAskill’s “Is It Good to Make Happy People?”

CarlShulman2y17

They're wildly quantitatively off. Straight 40% returns are way beyond equities, let alone the risk-free rate. And it's inconsistent with all sorts of normal planning, e.g. it would be against any savings in available investments, much concern for long-term health, building a house, not borrowing everything you could on credit cards, etc.

Similarly the risk aversion for rejecting a 15% of $1M for $1000 would require a bizarre situation (like if you needed just $500 more to avoid short term death), and would prevent dealing with normal uncertainty integral to life, like going on dates with new people, trying to sell products to multiple customers with occasional big hits, etc.

Brian_Tomasik

This page says: "The APRs for unsecured credit cards designed for consumers with bad credit are typically in the range of about 25% to 36%." That's not too far from 40%. If you have almost no money and would otherwise need such a loan, taking $100 now may be reasonable. There are claims that "Some 56% of Americans are unable to cover an unexpected $1,000 bill with savings", which suggests that a lot of people are indeed pretty close to financial emergency, though I don't know how true that is. Most people don't have many non-401k investments, and they roughly live paycheck to paycheck. I also think people aren't pure money maximizers. They respond differently in different situations based on social norms and how things are perceived. If you get $100 that seems like a random bonus, it's socially acceptable to just take it now rather than waiting for $140 next year. But it doesn't look good to take out big credit-card loans that you'll have trouble repaying. It's normal to contribute to a retirement account. And so on. People may value being normal and not just how much money they actually have. That said, most people probably don't think through these issues at all and do what's normal on autopilot. So I agree that the most likely explanation is lack of reflectiveness, which was your original point.

Critique of MacAskill’s “Is It Good to Make Happy People?”

CarlShulman2y26

Hi Brian,

I agree that preferences at different times and different subsystems can conflict. In particular, high discounting of the future can lead to forgoing a ton of positive reward or accepting lots of negative reward in the future in exchange for some short-term change. This is one reason to pay extra attention to cases of near-simultaneous comparisons, or at least to look at different arrangements of temporal ordering. But still the tradeoffs people make for themselves with a lot of experience under good conditions look better than what they tend to i... (read more)

Anthony DiGiovanni2y24

e.g. 2 minds with equally passionate complete enthusiasm (with no contrary psychological processes or internal currencies to provide reference points) respectively for and against their own experience, or gratitude and anger for their birth (past or future). They can respectively consider a world with and without their existences completely unbearable and beyond compensation. But if we're in the business of helping others for their own sakes rather than ours, I don't see the case for excluding either one's concern from our moral circle.
..

... (read more)

Brian_Tomasik2y12

Your reply is an eloquent case for your view. :)

This is one reason to pay extra attention to cases of near-simultaneous comparisons

In cases of extreme suffering (and maybe also extreme pleasure), it seems to me there's an empathy gap: when things are going well, you don't truly understand how bad extreme suffering is, and when you're in severe pain, you can't properly care about large volumes of future pleasure. When the suffering is bad enough, it's as if a different brain takes over that can't see things from the other perspective, and vice versa for... (read more)

Critique of MacAskill’s “Is It Good to Make Happy People?”

CarlShulman2y13

In the surveys they know it's all hypothetical.

You do see a bunch of crazy financial behavior in the world, but it decreases as people get more experience individually and especially socially (and with better cognitive understanding).

People do engage in rounding to zero in a lot of cases, but with lots of experience will also take on pain and injury with high cumulative or instantaneous probability (e.g. electric shocks to get rewards, labor pains, war, jobs that involve daily frequencies of choking fumes or injury.

Re lexical views that still make pr... (read more)

MichaelStJules

Yes, but they might not really be able to entertain the assumptions of the hypotheticals because they're too abstract and removed from the real world cases they would plausibly face. Very plausibly none of these possibilities would meet the lexical threshold, except with very very low probability. These people almost never beg to be killed, so the probability of unbearable suffering seems very low for any individual. The lexical threshold could be set based on bearableness or consent or something similar (e.g. Tomasik, Vinding). Coming up with a particular parameterization seems like a bit of work, though, and I'd need more time to think about that, but it's worth noting that the same practical problem applies to very large aggregates of finite goods/bads, e.g. Heaven or Hell, very long lives, or huge numbers of mind uploads. There's also a question of whether a life of unrelenting but less intense suffering can be lexically negative even if no particular experience meets some intensity threshold that would be lexically negative in all lives. Some might think of Omelas this way, and Mogensen's "The weight of suffering" is inclusive of this view (and also allows experiential lexical thresholds), although I don't think he discusses any particular parameterization.

The discount rate is not zero

CarlShulman2y8

I'd use reasoning like this, so simulation concerns don't have to be ~certain to drastically reduce EV gaps between local and future oriented actions.

The discount rate is not zero

CarlShulman2y8

Here's a good piece.

The discount rate is not zero

CarlShulman2y16

Today there is room for an intelligence explosion and explosive reproduction of AGI/robots (the Solar System can support trillions of AGIs for every human alive today). If aligned AGI undergoes such intelligence explosion and reproduction there is no longer free energy for rogue AGI to grow explosively. A single rogue AGI introduced to such a society would be vastly outnumbered and would lack special advantages, while superabundant AGI law enforcement would be well positioned to detect or prevent such an introduction in any case.

Already today states have r... (read more)

The discount rate is not zero

CarlShulman2y18

Barring simulation shutdown sorts of things or divine intervention I think more like 1 in 1 million per century, on the order of magnitude of encounters with alien civilizations. Simulation shutdown is a hole in the argument that we could attain such a state, and I think a good reason not to say things like 'the future is in expectation 50 orders of magnitude more important than the present.'

Vasco Grilo

8mo

Hi Carl, Simulation shutdown would end value in our reality, but in that of the simulators it would presumably continue, such that future expected value would be greater than the suggested by an extinction risk of 10^-6 per century? On the other hand, even if this is true, it would not matter because it would be quite hard to influence the eventual simulators?

RyanCarey

Whether simulation shutdown is a good reason not to say such things would seem to depend on how you model the possibility of simulation shutdown. One naive model would say that there is a 1/n chance that the argument that such a risk exists is correct, and if so, there is 1/m annual risk, otherwise, there is 0 annual risk from simulation shutdown. In such a model, the value of the future endowment would only be decreased n-fold. Whereas if you thought that there was definitely a 1/(mn) annual risk (i.e. the annual risk is IID) then that risk would diminish the value of the cosmic endowment by many OoM.

3[anonymous]2y

What is the source of the estimate of the frequency of encounters with alien civilizations?

The discount rate is not zero

CarlShulman2y103

It's quite likely the extinction/existential catastrophe rate approaches zero within a few centuries if civilization survives, because:

Riches and technology make us comprehensively immune to natural disasters.
Cheap ubiquitous detection, barriers, and sterilization make civilization immune to biothreats
Advanced tech makes neutral parties immune to the effects of nuclear winter.
Local cheap production makes for small supply chains that can regrow from disruption as industry becomes more like information goods.
Space colonization creates robustness agains

... (read more)

Tom_Davidson

Could you say more about what you mean by this?

mikbp

[Epistemic status: I have just read the summary of the post by the author and by Zoe Williams] Doesn't it make more sense to just set the discount rate dynamically accounting for the current (at the present) estimations? If/when we reach a point where the long-term existential catastrophe rate approaches zero, then it will be the moment to set the discount rate to zero. Now it is not, so the discount rate should be higher as the author proposes. Is there a reason for not using in a dynamic discount rate?

James Özden

This seems like an important claim, but I could also quite plausibly see misaligned AIs destabilising a world where an aligned AI exists. What reasons do we have to think that an aligned AI would be able to very consistently (e.g. 99.9%+ of the time) ward off attacks from misaligned AI from bad actors? Given all the uncertainty around these scenarios, I think the extinction risk per century from this alone could in the 1-5% range and create a nontrivial discount rate.

Jack Malde2y17

A classic paper by the climate economist Martin Weitzman shows that the average discount rate over long periods is set by the lowest plausible rate (as the possibilities of high rates drop out after a short period).

Not really disagreeing with anything you've said, but I just wanted to point out that this sentence doesn't quite acknowledge the whole story. To quote from Hilary Greaves' "Discounting for public policy: A survey" paper:

Weitzman did not, however, supply any fundamental justification for his assumption that the effective discount factor is

... (read more)

MichaelStJules2y10

What would be the per-century risk in such a state?

Also, does the >50% of such a state account for the possibility of alien civilizations destroying us or otherwise limiting our expansion?

Toby Ord’s The Scourge, Reviewed

CarlShulman2y7

An example of an argument matching the OP's thesis: Bryan Caplan rejecting animal rights (disagreeing with his favorite philosopher, Michael Huemer) based on the demands of applying a right to life to wild insects.