All of Pablo's Comments + Replies

In particular, superforecasters are generally more sceptical of such short AI timelines.

Note that this is true only of some subset of superforecasters. Samotsvety’s forecasters (many of whom are superforecasters) have shorter timelines than both domain experts and general x-risk experts:

 P(AGI by 2030)[4]P(AGI by 2050)P(AGI by 2100)P(AGI by this year) = 10%P(AGI by this year) = 50%P(AGI by this year) = 90%
mean:0.310.630.81202620412164
stdev:0.070.110.091.078.9979.65
       
50% CI:[0.26, 0.35][0.55, 0.70][0.74, 0.8
... (read more)
1
Zershaaneh Qureshi
10m
Thank you for this helpful response, Pablo! This is a really interesting result to note. I was not aware of this when writing my post, but I'll plan to include Samotsvety forecast results in future work on this subject, and when I come to write up my findings more formally. 

I guess most people sympathetic to existential risk reduction think the extinction risk from AI is much higher that those from other risks (as I do). In addition, existential risk as a fraction of extinction risk is arguably way higher for AI than other risks, so the consideration you mentioned will tend to make AI existential risk even more pressing? If so, people may be more interested in either tackling AI risk, or assessing its interactions with other risks.

Yes, this seems right.

As a semi-tangential observation: your comment made me better appreciate a... (read more)

I agree that in the absence of specific examples the criticism is hard to understand. But I would go further and argue that the NB at the beginning is fundamentally misguided and that well-meaning and constructive criticism of EA orgs or people should very rarely be obscured to make it seem less antagonistic.

Answer by PabloFeb 27, 202419
6
0

I would like someone write a post expanding on X-risks to all life v. to humans. Despite the importance of this consideration, it seems to have been almost completely neglected in EA discussion.

If I were to write on this, I’d reframe the issue somewhat differently than the author does in that post. Instead of a dichotomy between two types of risks, one could see it as a gradation of risks that push things back an increasing number of possible great filters. Risks to all life and risks to humans would then be two specific instances of this more general phen... (read more)

1
Dave Cortright
23d
I wrote this post asking what success for sentience looks like. There's a good chance we humans are just another stepping stone on the path toward an even higher form of intelligence and sentience.
6
Vasco Grilo
23d
Nice point, Pablo! I did not know about the post you linked, but I had noted it was not mentioned (at least very clearly) in Hilary Greaves' working paper Concepts of existential catastrophe[1] (at least in the version of September 2023) I guess most people sympathetic to existential risk reduction think the extinction risk from AI is much higher than those from other risks (as I do). In addition, existential risk as a fraction of extinction risk is arguably way higher for AI than other risks, so the consideration you mentioned will tend to make AI existential risk even more pressing? If so, people may be more interested in either tackling AI risk, or assessing its interactions with other risks. A related seemingly underexplored question is determining under which conditions human disempowerment (including extinction) would be bad from an impartial perspective. Humans have arguably played a role in the extinction of many species, including maybe some of the genus Homo (there are 13!), but that was not an existential risk given humans are thought to be better steerers of the future. The same might apply to AI under some conditions. Matthew Barnett has a quick take somewhat related to this. Here is the 1st paragraph: 1. ^ So I sent her an email a few days ago about this.

Thanks for the clarification.

Yes, I agree that we should consider the long-term effects of each intervention when comparing them. I focused on the short-term effects of hastening AI progress because it is those effects that are normally cited as the relevant justification in EA/utilitarian discussions of that intervention. For instance, those are the effects that Bostrom considers in ‘Astronomical waste’. Conceivably, there is a separate argument that appeals to the beneficial long-term effects of AI capability acceleration. I haven’t considered this argument because I haven’t seen many people make it, so I assume that accelerationist types tend to believe that the short-term effects dominate.

2
Matthew_Barnett
2mo
I think Bostrom's argument merely compares a pure x-risk (such as a huge asteroid hurtling towards Earth) relative to technological acceleration, and then concludes that reducing the probability of a pure x-risk is more important because the x-risk threatens the eventual colonization of the universe. I agree with this argument in the case of a pure x-risk, but as I noted in my original comment, I don't think that AI risk is a pure x-risk. If, by contrast, all we're doing by doing AI safety research is influencing something like "the values of the agents in society in the future" (and not actually influencing the probability of eventual colonization), then this action seems to plausibly just wash out in the long-term. In this case, it seems very appropriate to compare the short-term effects of AI safety to the short-term effects of acceleration. Let me put it another way. We can think about two (potentially competing) strategies for making the future better, along with their relevant short and possible long-term effects: * Doing AI safety research * Short-term effects: makes it more likely that AIs are kind to current or near-future humans * Possible long-term effect: makes it more likely that AIs in the very long-run will share the values of the human species, relative to some unaligned alternative * Accelerating AI * Short-term effect: helps current humans by hastening the arrival of advanced technology * Possible long-term effect: makes it more likely that we have a large population size at low per capita incomes, relative to a low population size with high per capita income My opinion is that both of these long-term effects are very speculative, so it's generally better to focus on a heuristic of doing what's better in the short-term, while keeping the long-term consequences in mind. And when I do that, I do not come to a strong conclusion that AI safety research "beats" AI acceleration, from a total utilitarian perspective.

I was trying to hint at prima facie plausible ways in which the present generation can increase the value of the long-term future by more than one part in billions, rather than “assume” that this is the case, though of course I never gave anything resembling a rigorous argument.

I do agree that the “washing out” hypothesis is a reasonable default and that one needs a positive reason for expecting our present actions to persist into the long-term. One seemingly plausible mechanism is influencing how a transformative technology unfolds: it seems that the firs... (read more)

4
Matthew_Barnett
2mo
As I understand, the argument originally given was that there was a tiny effect of pushing for AI acceleration, which seems outweighed by unnamed and gigantic "indirect" effects in the long-run from alternative strategies of improving the long-run future. I responded by trying to get more clarity on what these gigantic indirect effects actually are, how we can predictably bring them about, and why we would think it's plausible that we could bring them about in the first place. From my perspective, the shape of this argument looks something like: 1. Your action X has this tiny positive near-term effect (ETA: or a tiny direct effect) 2. My action Y has this large positive long-term effect (ETA: or a large indirect effect) 3. Therefore, Y is better than X. Do you see the flaw here? Well, both X and Y could have long-term effects! So, it's not sufficient to compare the short-term effect of X to the long-term effect of Y. You need to compare both effects, on both time horizons. As far as I can tell, I haven't seen any argument in this thread that analyzed and compared the long-term effects in any detail, except perhaps in Ryan Greenblatt original comment, in which he linked to some other comments about a similar topic in a different thread (but I still don't see what the exact argument is). More generally, I think you're probably trying to point to some concept you think is obvious and clear here, and I'm not seeing it, which is why I'm asking you to be more precise and rigorous about what you're actually claiming. In my original comment I pointed towards a mechanism. Here's a more precise characterization of the argument: 1. Total utilitarianism generally supports, all else being equal, larger population sizes with low per capita incomes over small population sizes with high per capita incomes.  2. To the extent that our actions do not "wash out", it seems reasonable to assume that pushing for large population sizes now would make it more likely in the long-ru

I think it remains the case that the value of accelerating AI progress is tiny relative to other apparently available interventions, such as ensuring that AIs are sentient or improving their expected well-being conditional on their being sentient. The case for focusing on how a transformative technology unfolds, rather than on when it unfolds,[1] seems robust to a relatively wide range of technologies and assumptions. Still, this seems worth further investigation.

  1. ^

    Indeed, it seems that when the transformation unfolds is primarily important because of

... (read more)
6
Matthew_Barnett
2mo
I'm claiming that it is not actually clear that we can take actions that don't merely wash out over the long-term. In this case, you cannot simply assume that we can meaningfully and predictably affect how valuable the long-term future will be in, for example, billions of years. I agree that, yes, if you assume we can meaningfully affect the very long-run, then all actions that merely have short-term effects will have "tiny" impacts by comparison. But the assumption that we can meaningfully and predictably affect the long-run is precisely the thing that needs to be argued. I think it's important for EAs to try to be more rigorous about their empirical claims here. Moreover, actions that have short-term effects can generally be assumed to have longer term effects if our actions propagate. For example, support for larger population sizes now would presumably increase the probability that larger population sizes exist in the very long run, compared to the alternative of smaller population sizes with high per capita incomes. It seems arbitrary to assume this effect will be negligible but then also assume other competing effects won't be negligible. I don't see any strong arguments for this position.

Recently, an OP grantee spent 5 hours of staff time opening this Chase Platinum Business Checking account which swept into this JPMorgan US Treasury Plus Money Market Fund.

I tried to open a Chase Platinum Business Checking account for a 501(c)(3) that I created recently, but it appears that they are not available for nonprofits. If one selects “Corporation” in the dropdown menu on the left, one is forced to select either “S-Corporation” or “C-Corporation” in the dropdown menu on the right, neither of which, I believe, is appropriate for a nonprof... (read more)

Thank you for the ping; I’ll take a look shortly.

Great post. A few months I wrote a private comment that makes a very similar point but frames it somewhat differently; I share it below in case it is of any interest.

Victoria Krakovna usefully defines the outer and inner alignment problems in terms of different “levels of specification”: the outer alignment problem is the problem of aligning the ideal specification (the goals of the designer) with the design specification (the goal implemented in the system), while the inner alignment problem is the problem of aligning this design specification with the re

... (read more)
4
Matthew_Barnett
3mo
I replied to your comment in a new post here.

This deck includes some EAA-related numbers, which may be of interest.

Pablo
4mo50
14
0
4

Retaliation is bad.

People seem to be using “retaliation” in two different senses: (1) punishing someone merely in response to their having previously acted against the retaliator’s interests, and (2) defecting against someone who has previously defected in a social interaction analogous to a prisoner’s dilemma, or in a social context in which there is a reasonable expectation of reciprocity. I agree that retaliation is bad in the first sense, but Will appears to be using ‘retaliation’ in the second sense, and I do not agree that retaliation is bad in this ... (read more)

Humanity’s chances of realizing its potential are substantially lower when there are only a few thousand humans around, because the species will remain vulnerable for a considerable time before it fully recovers. The relevant question is not whether the most severe current risks will be as serious in this scenario, because (1) other risks will then be much more pressing and (2) what matters is not the risk survivors of such a catastrophe face at any given time, but the cumulative risk to which the species is exposed until it bounces back.

Two straightforward ways (more have been discussed in the relevant literature) are by making humanity more vulnerable to other threats and by pushing back humanity past the Great Filter (about whose location we should be pretty uncertain).

2
titotal
4mo
This is very vague. What other threats? It seems like a virus wiping out most of humanity would decrease the likelihood of other threats. It would put an end to climate change, reduce the motivation for nuclear attacks and ability to maintain a nuclear arsenal, reduce the likelihood of people developing AGI, etc.  

The relevant comparison in this context is not with human extinction but with an existential catastrophe. A virus that killed everyone except humans in extremely remote locations might well destroy humanity’s long-term potential. It is not plausible—at least not for the reasons provided— that “GCR's may be many, many orders of magnitude more likely than” existential catastrophes, on reasonable interpretations of “many, many”.

(Separately, the catastrophe may involve a process that intelligently optimizes for human extinction, by either humans or non-human agents, so I also think that the claim as stated is false.)

4
titotal
4mo
How? I see it delaying things while the numbers recover, but it's not like humans will suddenly become unable to learn to read. Why would humanity not simply pick itself up and recover? 

Scott Alexander's writing style is also worthy of broader emulation among EAs

I very much agree. Here’s a post by Scott with nonfiction writing advice.

The Summary of posts in the sequence alone was super useful. Perhaps the RP team would like to include it, or a revised version of it, in the sequence introduction?

Thanks for the idea, Pablo. I've added summaries to the sequence page.

Cool. I'd be interested in tentatively providing this search for free on EA News via the OpenAI API, depending on monthly costs. Do you know how to implement it?

1
Daniel_Friedrich
4mo
Sorry, I don't have any experience with that.

Very interesting. I’d personally appreciate a full post.

The board’s behavior was grossly unprofessional

You had no evidence to justify that claim back when you made it, and as new evidence is released, it looks increasingly likely that the claim was not only unjustified but also wrong (see e.g. this comment by Gwern).

Does anyone know where I can find something like that?

You can take a look at the ‘Further reading’ section of criticism of effective altruism, the articles so tagged, and the other tags starting with “criticism of” in the ‘Related entries’ section.

I'm not sure how to react to all of this, though.

Kudos for being uncertain, given the limited information available.

(Not something one cay say about many of the other comments to this post, sadly.)

I still tend to agree the expected value of the future is astronomical (e.g. at least 10^15 lives), but then the question is how easily one can increase it.

If one grants that the time of perils will last at most only a few centuries, after which the per-century x-risk will be low enough to vindicate the hypothesis that the bulk of expected value lies in the long-term (even if one is uncertain about exactly how low it will drop), then deprioritizing longtermist interventions on tractability grounds seems hard to justify, because the concentration of total x-risk in the near-term means it's comparatively much easier to reduce.

2
Vasco Grilo
4mo
I am not sure proximity in time is the best proxy for tractability. The ratio between the final and current global GDP seems better, as it accounts for both the time horizon, and rate of change/growth over it. Intuitively, for a fixed time horizon, the higher the rate of change/growth, the harder it is to predict the outcomes of our actions, i.e. tractability will tend to be lower. The higher tractability linked to a short time of perils may be roughly offset by the faster rate of change over it. Maybe Aschenbrenner's paper on existential risk and growth can inform this? Note I am quite sympathetic to influencing the longterm future. As I said, I have been donating to the LTFF. However, I would disagree that donating to the LTFF is astronomically (e.g. 10 OOMs) better than to the Animal Welfare Fund.

Hi Vasco,

I can see the above applying for some definitions of time of perils and technological maturity, but then I think they may be astronomically unlikely.

What do you think about these considerations for expecting the time of perils to be very short in the grand scheme of things? It just doesn't seem like the probability of possible future scenarios decays nearly fast enough to offset their greater value in expectation.

3
Vasco Grilo
4mo
Hi Pablo, Those considerations make sense to me, but without further analysis it is not obvious to me whether they imply e.g. an annual existential risk in 2300 of 0.1 % or 10^-10, or e.g. a longterm existential risk of 10^-20 or 10^-60. I still tend to agree the expected value of the future is astronomical (e.g. at least 10^15 lives), but then the question is how easily one can increase it.

Semi-tangential question: what's the rationale for making the reactions public but the voting (including the agree/disagree voting) anonymous?

Here’s an article by @Brian_Tomasik enumerating, and briefly discussing, what strike me as most of the relevant considerations for and against publishing in academia.

The ‘citability’ consideration also applies to Wikipedia, which requires that all claims be supported by “reliable sources” (and understands that notion quite narrowly). For example, many concepts developed by the rationalist and EA communities cannot be the subject of a Wikipedia article merely because they have not received coverage in academic publications.

There are a handful of bottlenecks and a number of ways to decompose this problem.  To get started on subproblems, one of us (Ethan) is working on a list of suggestions, which we will backlink here.

@eca, did you end up writing this list?

I don't understand your underlying model of human psychology. Sam Bankman-Fried was super-rich and powerful, but is now the kind of person no one would touch with the proverbial ten-foot pole. If the claim is that humans tend to like super-rich and powerful people even after they become disgraced, that seems false based on informal evidence.

In any case, from what I know about Bankman-Fried and his actions, the claim that he did not knowingly steal customer money doesn't strike me as obviously false, and in line with my sense that much of his behavior is explained by a combination of gross incompetence and pathological delusion.

2
DC
6mo
  I think you fail to empathize with aspects of the nature of power, particularly in that there is a certain fraction of humans who will find cachet in the edgy and criminal. I am not that surprised Lewis may have been unduly affected by being in Sam's orbit and getting front-row seats to such a story. Though for all I know maybe he has accurate insider info, and Sam actually didn't knowingly steal money. ¯\_(ツ)_/¯

Effective Altruism News also offers this: just click ‘Run this search externally’ after typing your search query. We currently index ~350 sites.

Thanks for the feedback! Although I am no longer working on this project, I am interested in your thoughts because I am currently developing a website with Spanish translations, which will also feature a system where each tag is also a wiki article and vice versa. I do think that tags and wiki articles have somewhat different functions and integrating them in this way can sometimes create problems. But I'm not sure I agree that the right approach is to map multiple tags onto a single article. In my view, a core function of a Wiki is to provide concise defi... (read more)

But if they both use microdooms, they can compare things 1:1 in terms of their effect on the future, without having to flesh out all of the post-agi cruxes.

I don't think this is the case for all key disagreements, because people can disagree a lot about the duration of the period of heightened existential risk, whereas microdooms are defined as a reduction in total existential risk rather than in terms of per-period risk reduction. So two people can agree that AI safety work aimed at reducing existential risk will decrease risk by a certain amount over a g... (read more)

“the attainment of capabilities affording a level of economic productivity and control over nature close to the maximum that could feasibly be achieved.” (Nick Bostrom (2013) ‘Existential risk prevention as global priority’, Global Policy, vol. 4, no. 1, p. 19.)

Loneliness in a house share situation depends entirely on whether your housemates are good, and there is no guarantee that your co-workers are good housemates just because they are EA.

The most plausible version of this argument is not that someone will be a good housemate just because they are EA. It is that banning or discouraging EA co-living makes it more difficult for people to find any co-living arrangement.

Thanks for the update: I have retracted the relevant part of my previous comment.

Talking with some physicist friends helped me debunk the many worlds thing Yud has going.

Yudkowsky may be criticized for being overconfident in the many-worlds interpretation, but to feel that you have “debunked” it after talking to some physicist friends shows excessive confidence in the opposite direction. Have you considered how your views about this question would have changed if e.g. David Wallace had been among the physicists you talked to?

Also, my sense is that “Yud” was a nickname popularized by members of the SneerClub subreddit (one of the most i... (read more)

1
Jonas Hallgren
7mo
Sorry, Pablo, I meant that I got a lot more epistemically humble, I should have thought about how I phrased it more. It was more that I went from the opinion that many worlds is probably true to: "Oh man, there are some weird answers to the Wigner's friend thought experiment and I should not give a major weight to any." So I'm more like maybe 20% on many worlds?  That being said I am overconfident from time to time and it's fair to point that out from me as well. Maybe you were being overconfident in saying that I was overconfident? :D
4
Devin Kalish
7mo
I don’t have a link because Twitter is very difficult to search now if you don’t have an account (if someone wants to provide one be my guest, there’s one discussion thread involving Zach Weinersmith that says so for instance), but Yudkowsky currently uses and seems to like the nickname at this point.

I'm very excited to see this.

One minor correction: Toby Ord assigns a 1-in-6 chance of existential catastrophe this century, which isn't equivalent to a 1-in-6 chance of human extinction.

1
dominicroser
7mo
oh dear. Thanks for spotting this - really appreciated. We'll add the correction.

Indeed. And there are other forecasting failures by Mearsheimer, including one in which he himself apparently admits (prior to resolution) that such a failure would constitute a serious blow to his theory. Here’s a relevant passage from a classic textbook on nuclear strategy:[1]

In an article that gained considerable attention, largely for its resolute refusal to share the general mood of optimism that surrounded the events of 1989, John Mearsheimer assumed that Germany would become a nuclear power. Then, as the Soviet Union collapsed, he explained why it m

... (read more)

In my view, the comment isn't particularly responsive to the post.

Shouldn’t we expect people who believe that a comment isn’t responsive to its parent post to downvote it rather than to disagree-vote it, if they don’t have any substantive disagreements with it?

I am puzzled that, at the time of writing, this comment has received as many disagreement votes as agreement votes. Shouldn't we all agree that the EA community should allocate significantly more resources to an area, if by far the most good can be done by this allocation and there are sound public arguments for this conclusion? What are the main reasons for disagreement?

  1. Different people in EA define 'good' in different ways. You can argue that some cause is better for some family of definitions, but the aim is, I think, to help people with different definitions too achieve the goal.
  2. You say "if by far the most good can be done by this allocation and there are sound public arguments for this conclusion", but the idea of 'sound public arguments' is tricky. We're not scientists with some very-well-tested models. You're never going to have arguments which are conclusive enough to shut down other causes, even if it sometimes seems to some people here that they do.
8
Jason
8mo
In my view, the comment isn't particularly responsive to the post. I take the post's main critique as being something like: groups present themselves as devoted to EA as a question and to helping participants find their own path in EA, but in practice steer participants heavily toward certain approved conclusions. That critique is not inconsistent with "EA resources should be focused on AI and longtermism," or maybe even "EA funding for university groups should concentrate on x-risk/AI groups that don't present themselves to be full-spectrum EA groups."

To be clear, I was only trying to describe what I believe is going on here, without necessarily endorsing the relative neglect of this cause area. And it does seem many EA folk consider radical life extension a “speculative” way of improving global health, whether or not they are justified in this belief. 

Welcome to the Forum!

I think part of the explanation relates to what you point out at the end: “radical life extension is speculative”. You note that “so is preventing risks from AI”, but preventing such risks seems to be higher impact than extending life. In general, causes vary both in how “speculative” they are and in what their expected impact is, and EA may be seen as an attempt to have the most impact for varying levels of “speculativeness”. One could argue that, while the impact of life extension as a cause is relatively high, its “speculativeness-a... (read more)

1
D0TheMath
8mo
This seems false. Dramatic increases in life extension technology have been happening ever since the invention of modern medicine, so its strange to say the field is speculative enough not to even consider.

Having listened to several episodes, I can strongly recommend this podcast. One of the very best.

If "Eugenics-Adjacent" is not Torres but tipped off Torres about the article, that also seems like a good reason for downvoting the post, since it indicates that the username was chosen to cause damage to EA rather than to stimulate an honest discussion.

I can confirm it's open access. I know this because my team is translating it into Spanish and OUP told us so. Our aim is to publish the Spanish translation simultaneously with the English original (the translation will be published online, not in print form).

A recurrent disappointment with EA Forum comments related to the karma system is the frequency with which they assume that a particular explanation of the karma score of a given comment is correct, without even considering alternative explanations. One doesn't have to be very creative or contrarian to think of plausible reasons users may have disagreed-voted with the statement "CEA is widely perceived as grossly ineffective whereas AMF is perceived as highly effective" other than that "CEA staff have so much karma that they can tank any comment with a single downvote".

I'm puzzled by this reply. You acknowledge that you misremembered, but in your edited comment you continue to state that "here's a whole chapter in superintelligence on human intelligence enhancement via genetic selection".

The chapter in question is called "Paths to superintelligence", and is divided into five sections, followed by a summary. The sections are called "Artificial intelligence", "Whole brain emulation", "Biological cognition", "Brain–computer interfaces" and "Networks and organizations". It is evident that the whole chapter is not devoted to ... (read more)

Note that this is a different Peter Singer. Here is the entry for the relevant Peter Singer.  According to it, Singer had 77 co-authors.

4
Ben_West
3mo
Oops, good correction, thanks!

Thanks, I'll remove it next time I update the deck.

Load more