All of Rocket&#x27;s Comments + Replies

All AGI Safety questions welcome (especially basic ones) [May 2023]

Suppose someone is an ethical realist: the One True Morality is out there, somewhere, for us to discover. Is it likely that AGI will be able to reason its way to finding it?

What are the best examples of AI behavior we have seen where a model does something "unreasonable" to further its goals? Hallucinating citations?

Rocket1y3

What are the arguments for why someone should work in AI safety over wild animal welfare? (Holding constant personal fit etc)

If someone thinks wild animals live positive lives, is it reasonable to think that AI doom would mean human extinction but maintain ecosystems? Or does AI doom threaten animals as well?
Does anyone have BOTECs on numbers of wild animals vs numbers of digital minds?

Maybe Family Planning Charities Are Better For Farmed Animals Than Animal Welfare Ones

Predictable updating about AI risk

At least we can have some confidence in the total weight of meat consumed on average by a Zambian per year and the life expectancy at birth in Zambia.

We should also think about these on the margin. Ie the lives averted might have been shorter than average and consumed less meat than average.

Hank_B

That's true! Maybe the potential human would have been born to poorer than average parents (because those are the people who need help accessing contraception), thus being poorer on average (and so consuming less meat). Or maybe the potential human would be born to more educated on average parents (since those are the people who'd be interested in using contraception?)? Thus being richer on average and eating more meat.

Talking publicly about AI risk

I imagine a proof (by contradiction) would work something like this:

Suppose you place > 1/x probability on your credences moving by a factor of x. Then the expectation of your future beliefs is > prior * x * 1/x = prior, so your credence will increase. With our remaining probability mass, can we anticipate some evidence in the other direction, such that our beliefs still satisfy conservation of expected evidence? The lowest our credence can go is 0, but even if we place our remaining < 1 - 1/x probability on 0, we would still find future beliefs &... (read more)

What concrete question would you like to see debated? I might organise some debates.

I would endorse all of this based on experience leading EA fellowships for college students! These are good principles not just for public media discussions, but also for talking to peers.

Rocket1y2

Yay! Where in the Bay are you located?

Answer by RocketApr 20, 20231

THW double the size of EA.

What concrete question would you like to see debated? I might organise some debates.

Answer by RocketApr 20, 20232

THB that EA-minded college freshmen should study Computer Science over Biology

Effective Altruism's Implicit Epistemology

Effective Altruism's Implicit Epistemology

Thanks for the thorough response! I agree with a lot of what you wrote, especially the third section on Epistemic Learned Helplessness: "Bayesianism + EUM, but only when I feel like it" is not a justification in any meaningful sense.

On Priors

I agree that we can construct thought experiments (Pascal's Mugging, acausal trade) with arbitrarily high stakes to swamp commonsense priors (even without religious scenarios or infinite value, which are so contested I think it would be difficult to extract a sociological lesson from them).

On Higher Order Evidence

I sti... (read more)

Rocket1y4

Thanks for the excellent post!

I think you are right that this might be a norm/heuristic in the community, but in the spirit of a "justificatory story of our epistemic practices," I want to look a little more at

4. When arguments lead us to conclusions that are both speculative and fanatical, treat this as a sign that something has gone wrong.

First, I'm not sure that "speculative" is an independent reason that conclusions are discounted, in the sense of a filter that is applied ex-post. In your 15AI thought experiment, for example, I think ... (read more)

Violet Hour

Thanks for the comment! (Fair warning, my response will be quite long) I understand you to be offering two potential stories to justify ‘speculativeness-discounting’. 1. First, EAs don’t (by and large) apply a speculativeness-discount ex post. Instead, there’s a more straightforward ‘Bayesian+EUM’ rationalization of the practice. For instance, the epistemic practice of EAs may be better explained with reference to more common-sense priors, potentially mediated by orthodox biases. 2. Or perhaps EAs do apply a speculativeness-discount ex post. This too can be justified on Bayesian grounds. 1. We often face doubts about our ability to reason through all the relevant considerations, particularly in speculative domains. For this reason, we update on higher-order uncertainty, and implement heuristics which themselves are justified on Bayesian grounds. In my response, I’ll assume that your attempted rationale for Principle 4 involves justifying the norm with respect to the following two views: * Expected Utility Maximization (EUM) is the optimal decision-procedure. * The relevant probabilities to be used as inputs into our EUM calculation are our subjective credences. The ‘Common Sense Priors’ Story I think your argument in (1) is very unlikely to provide a rationalization of EA practice on ‘Bayesian + EUM’ grounds.[1] Take Pascal’s Mugging. The stakes can be made high enough that the value involved can easily swamp your common-sense priors. Of course, people have stories for why they shouldn’t give the money to the mugger. But these stories are usually generated because handing over their wallet is judged to be ridiculous, rather than the judgment arising from an independent EU calculation. I think other fanatical cases will be similar. The stakes involved under (e.g.) various religious theories and our ability to acausally affect an infinite amount of value are simply going to be large enough to swamp our initial common-sense priors. Thus, I think

Some Things I Heard about AI Governance at EAG

What to think when a language model tells you it's sentient

Copy that. I removed "smash," but I'm leaving the language kind of ambiguous because my understanding of this strategy is that it's not restricted to conventional regulations, but instead will draw on every available tool, including informal channels.

Rocket1y6

Thanks for following up and thanks for the references! Definitely agree these statements are evidence; I should have been more precise and said that they're weak evidence / not likely to move your credences in the existence/prevalence of human consciousness.

What to think when a language model tells you it's sentient

Rocket1y2

a very close connection between an entity’s capacity to model its own mental states, and consciousness itself.

The 80k episode with David Chalmers includes some discussion of meta-consciousness and the relationship between awareness and awareness of awareness (of awareness of awareness...). Would recommend to anyone interested in hearing more!

They make the interesting analogy that we might learn more about God by studying how people think about God than by investigating God itself. Similarly we might learn more about consciousness by investigating how people think about it...

rgb

Agree, that's a great pointer! For those interested, here is the paper and here is the podcast episode. [Edited to add a nit-pick: the term 'meta-consciousness' is not used, it's the 'meta-problem of consciousness', which is the problem of explaining why people think and talk the way they do about consciousness]

What to think when a language model tells you it's sentient

Rocket1y6

We trust human self-reports about consciousness, which makes them an indispensable tool for understanding the basis of human consciousness (“I just saw a square flash on the screen”; “I felt that pinprick”).

I want to clarify that these are examples of self-reports about consciousness and not evidence of consciousness in humans. A p-zombie would be able to report these stimuli without subjective experience of them.

They are "indispensable tools for understanding" insofar as we already have a high credence in human consciousness.

rgb

Thanks for the comment. A couple replies: Self-report is evidence of consciousness in Bayesian sense (and in common parlance): in a wide range of scenarios, if a human says they are conscious of something, you should have a higher credence than if they do not say they are. And in the scientific sense: it's commonly and appropriately taken as evidence in scientific practice; here is Chalmers's "How Can We Construct a Science of Consciousness?" on the practice of using self-reports to gather data about people's conscious experiences: It's suppose it's true that self-reports can't budge someone from the hypothesis that other actual people are p-zombies, but few people (if any) think that. From the SEP: So yeah: my take is that no one, including anti-physicalists who discuss p-zombies like Chalmers, really thinks that we can't use self-report as evidence, and correctly so.

JWS

This is true for literally all empirical evidence if you accept the possibility of a P-Zombie. The only possible falsification for consciousness can come from the internal subject itself, nothing else will do. But for everyone apart from you, it's self-reports, 3rd party observation, or nothing. Edit: What I mean here is that these self-reports are evidence - if they're not then there's no evidence for any minds apart from your own. And therefore we also ought to take AI self-reports as evidence. Not as serious as we take human self-reports at this stage, but evidence nonetheless.

CEA statement on Nick Bostrom's email

Rocket1y0

Another statement that "people are equal" from GWWC.

CEA statement on Nick Bostrom's email

Longtermism and Animal Farming Trajectories

EMs?

Habryka

"Emulated Minds" aka "Mind uploads".

Larks

Brain Emulations - basically taking a person and running a simulation of them on a computer, where they could potentially be copied, run faster or slower, etc.

Rocket1y2

Thanks for this post! I'm wondering what social change efforts you find most promising?

MichaelDello

Thanks for the great question. I'd like to see more attempts to get legislation passed to lock in small victories. The Sioux Falls slaughterhouse ban almost passing gives me optimism for this. Although it seemed to be more for NIMBY reasons than for animal rights reasons, in some ways that doesn't matter. I'm also interested in efforts to maintain the lower levels of speciesism we see in children into their adult lives, and to understand what exactly drives that so we can incorporate it into outreach attempts targeted at adults. Our recent interview with Matti Wilks touches on this a little if you're interested.

From less than a week ago: https://forum.effectivealtruism.org/posts/PPeiDDD7PgQA35RgF/apply-for-economics-phd-application-mentorship-and-support

Oh I see! Ya, crazy stuff. I liked the attention it paid to the role of foundation funding. I've seen this critique of foundations included in some intro fellowships, so I wonder if it would also especially resonate with leftists who are fed up with cancel culture in light of the Intercept piece.

I don't think anything here attempts a representation of "the situation in leftist orgs" ? But yes lol same

Linch

I don't know what you mean by "anything here," but I'm referring to the link that Larks shared.

https://forum.effectivealtruism.org/posts/MCuvxbPKCkwibpcPz/how-to-talk-to-lefties-in-your-intro-fellowship?commentId=YwQme9B2nHoH6fXeo

https://forum.effectivealtruism.org/posts/MCuvxbPKCkwibpcPz/how-to-talk-to-lefties-in-your-intro-fellowship?commentId=YwQme9B2nHoH6fXeo

https://forum.effectivealtruism.org/posts/MCuvxbPKCkwibpcPz/how-to-talk-to-lefties-in-your-intro-fellowship?commentId=YwQme9B2nHoH6fXeo

Rocket2y11

This is a response to D0TheMath, quinn, and Larks, who all raise some version of this epistemic concern:

(1) Showing how EA is compatible with leftist principles requires being disingenuous about EA ideas —> (2) recruit people who join solely based on framing/language —> (3) people join the community who don't really understand what EA is about —> (4) confusion!

The reason I am not concerned about this line of argumentation is that I don't think it attends to the ways people decide whether to become more involved in EA.

(2) In my experience, people a... (read more)

Rocket2y11

Ya maybe if your fellows span a broad political spectrum, then you risk alienating some and you have to prioritize. But the way these conversations actually go in my experience is that one fellow raises an objection, eg "I don't trust charities to have the best interests of the people they serve at heart." And then it falls to the facilitator to respond to this objection, eg "yes, PlayPumps illustrates this exact problem, and EA is interested in improving these standards so charities are actually accountable to the people they serve," etc.

My sense is... (read more)

david_reinstein

That makes sense in that context. Still I think that generally bringing in people to EA under the pretence that it is substantially lefty in these ways and accommodating to this style of discourse could have possibly negative consequences. If these people join and use this language in explaining EA to others it might end up turning others off.

utilistrutil's Quick takes

I agree with quinn. I'm not sure what the mechanism is by which we end up with lowered epistemic standards. If an intro fellow is the kind of person who weighs reparative obligations very heavily in their moral calculus, then deworming donations may very well satisfy this obligation for them. This is not an argument that motivates me very much, but it may still be a true argument. And making true arguments doesn't seem bad for epistemics? Especially at the point where you might be appealing to people who are already consequentialists, just consequentialists with a developed account of justice that attends to reparative obligations.

Some potential lessons from Carrick’s Congressional bid

Thanks for the reply! I'm satisfied with your answer and appreciate the thought you've put into this area :) I do have a couple follow-ups if you have a chance to share further:

I expected to hear about the value of the connections made at EAG, but I'm not sure how to think about the counterfactual here. Surely some people choose to meet up at EAG but in the absence of the conference would have connected virtually, for example?

I also wonder about the cause areas of the EA-aligned orgs you cited. Ie, I could imagine longtermist orgs that are more talen... (read more)

Eli_Nathan

No problem! I probably won't be able to respond to your later points, just because the answers would be complicated and I'd have to go into a lot of detail re how I think about EAG. But to answer some of your other questions: 1. I don't have concrete data on the counterfactual likelihood of connections, but I expect that it's not that high (very strong confidence that it's <50% of connections). There's no obvious way for many these people to connect virtually, other than attending a virtual EA conference, and I think there are also strong benefits to meeting in person (as well as the possibility of group discussions and meetups). My rough guess would also be that people in general are less interested in virtual conferences than in-person ones, meaning that there are a bunch of counterfactual connections here. 2. The org that said they’d gotten a minimum of $1.25 million worth of value from connections they’ve made at EAG(x)’s was a global health and development org. I don't know exactly who said that they would trade $5 million in donations for the contacts they made at EAGxBoston, but my guess is that this was someone working in a longtermist/x-risk field (someone on my team told me about this feedback, I didn't receive it directly myself).

utilistrutil's Quick takes

The second point implies more of a bright line than scalar dynamic, which seems consistent with scope insensitivity over lower donation amounts. That is, we might expect scope insensitivity to equalize the perception of $1m and $5m dollars, but once you hit $10m, then you attract negative media coverage. If we restrict ourselves to donation sizes that allow us to fly under the radar of national media outlets, then the scope insensitivity argument may still bite.