OCB

Owen Cotton-Barratt

8437 karmaJoined Aug 2014

Sequences
2

On Wholesomeness
Everyday Longermism

Comments
755

Topic contributions
3

I think you're missing some important ground in between "reflection process" and "PR exercise".

I can't speak for EV or other people then on the boards, but from my perspective the purpose of the legal investigation was primarily about helping to facilitate justified trust. Sam had by many been seen as a trusted EA leader, and had previously been on the board of CEA US. It seemed it wouldn't be unreasonable if people in EA (or even within EV) started worrying that leadership were covering things up. Having an external investigation was, although not a cheap signal to send, much cheaper in worlds where there was nothing to hide compared to worlds where we wanted to hide something. Between that and wanting to be able to credibly signal to external stakeholders like the Charity Commission, I think general PR was a long way down the list of considerations.

I haven't downvoted, but this is attracting several downvotes, and I thought I'd try to articulate some negative feelings I have here:

  • First, as Stefan has noted, the summary seems inaccurate: Zach's article nowhere claims that Sam was never an effective altruist
    • I think it's bad form to put sensationalist takeaways in a summary when they don't appear in the article, and feel not great about the link as a result
    • I do think that seeing it linked in this way primed me to be more negative about the article (and the notes below reflect that)
      • I have mixed feelings about that (maybe it would have been better to see it without anchoring, but also there's something good about looking at things we love with critical eyes)
  • Second cluster, some negativity towards the article
    • Could be summarized as "Except ... it's kind of right, though?" of the summary linked here. While the article never says Sam was never an EA in so many words, that's kind of the vibe that's carried through it
    • It doesn't acknowledge the (IMO real) possibility that Sam's errors were in part inspired by misguided readings of effective altruism
      • The judge slammed Sam in part for "evasive, hairsplitting" testimony and "never a word of remorse"
        • It seems like maybe Zach's article errs just a little in these directions (although far less egregiously, and certainly not lying or covering up any crime)
        • I have some feeling like it may be appropriate to go out of the way now (and especially on this topic) to try not to make any of Sam's mistakes
    • I have mixed feelings about the impact I expect this to have on readers
      • I think it says a lot of true things and I think that if taken at face value will make many people's impressions more accurate
      • However, I also think some proportion of insightful readers may pick up on some PR-vibes and have this reinforce an impression that effective altruism may still be vulnerable to some of the errors that Sam made
      • Probably it's good overall, compared to no such article? But I wonder if there might have been some version of the article that I'd have felt more straightforwardly good about

I guess that many of the downvotes are likely based on one or the other of these clusters (or both, although the two complaints are in some tension with one another).

(edited for clarity and to restructure the bullet point nesting)

4 is a great point, thanks.

On 1--3, I definitely agree that I may prudentially prefer some possibilities than others. I've been assuming that from a consequentialist moral perspective the distribution of future outcomes still looks like the one I give in this post, but I guess it should actually look quite different. (I think what's going on is that in some sense I don't really believe in world A, so haven't explored the ramifications properly.)

I support people poking at the foundations of these arguments. And I especially appreciated the discussion of bottlenecks, which I think is an important topic and often brushed aside in these discussions.

That said, I found that this didn't really speak to the reasons I find most compelling in favour of something like the singularity hypothesis. Thorstad says in the second blog post:

If each doubling of intelligence is harder to bring about than the last, then even if all AI research is eventually done by recursively self-improving AI systems, the pace of doubling will steadily slow.

I think this is wrong. (Though the paper itself avoids making the same mistake.) There are lots of coherent models where the effective research output of the AI systems is growing faster than the difficulty of increasing intelligence, leading to accelerating improvements despite each doubling of intelligence getting harder than the last. These are closely analogous to the models which can (depending on some parameter choices) produce a singularity in economic growth by assuming endogenous technological growth.

In general I agree with Thorstad that the notion of "intelligence" is not pinned down enough to build tight arguments on it. But I think that he goes too far in inferring that the arguments aren't there. Rather I think that the strongest versions of the arguments don't directly route through an analysis of intelligence, but something more like the economic analysis. If further investments in AI research drive the price-per-unit-of-researcher-year-equivalent down in fast enough, this could lead to hyperbolic increases in the amount of effective research progress, and this could in turn lead to rapid increases in intelligence -- however one measures that. I agree that this isn't enough to establish that things will be "orders of magnitude smarter than humans", but for practical purposes the upshot that "there will be orders of magnitude more effective intellectual labour from AI than from humans" does a great deal of work.

On the argument that extraordinary claims require extraordinary evidence, I'd have been interested to see Thorstad's takes on the analyses which suggest that long-term historical growth rates are hyperbolic, e.g. Roodman (2020). I think of that as one of the more robust long-term patterns in world history. The hypothesis which says "this pattern will approximately continue" doesn't feel to me to be extraordinary. You might say "ah, but that doesn't imply a singularity in intelligence", and I would agree -- but I think that if you condition on this kind of future hyperbolic growth in the economy, the hypothesis that there will be a very large accompanying increase in intelligence (however that's measured) also seems kind of boring rather than extraordinary.

Although I agree that all of these are challenges, I don't really believe they're enough to undermine the basic case. It's not unusual in high-paying industries for some people to make several times as much as their colleagues. So there's potentially lots of room to have higher consumption than you would working at a nonprofit while also giving away more than half your salary.

Empirically my impression is also that people who went into earning to give early tended to stay giving later in their careers 10+ years later; although that's anecdotal rather than data-driven.

I agree with "pretty rationally overall" with respect to general world modelling, but I think that some of the stuff about how it relates to its own values / future selves is a bit of a different magisterium and it wouldn't be too surprising if (1) it hadn't been selected for rationality/competence on this dimension, and (2) the general rationality didn't really transfer over.

One thought is that for something you're describing as a minimal viable takeover AI, you're ascribing it a high degree of rationality on the "whether to wait" question.

By default I'd guess that minimal viable takeover systems don't have very-strong constraints towards rationality. And so I'd expect at least a bit of a spread among possible systems -- probably some will try to break out early whether or not that's rational, and likewise some will wait even if that isn't optimal.

That's not to say that it's not also good to ask what the rational-actor model suggests. I think it gives some predictive power here, and more for more powerful systems. I just wouldn't want to overweight its applicability.

Yeah I'm arguing that with good reflective governance we should achieve a large fraction of what's accessible.

It's quite possible that that means "not quite all", e.g. maybe there are some trades so that we don't aestivate in this galaxy, but do in the rest of them; but on the aggregative view that's almost as good as aestivating everywhere.

I think in this context the natural way to interpret probabilities (or medians) of probabilities is as something like: "what would I think the probability was if I knew a lot more about the situation and got to think about it a lot more, but wasn't literally omniscient". Of course that isn't fully defined (since it's not specified exactly how much more they get to know), but I think it's approximately meaningful, and can capture something quite useful/important about credal resilience.

Relatedly, I think the journalist's presentation is misleading. I think it makes it sound like there's something like an objective 1-in-230 chance of catastrophe. I think a more accurate presentation would be "expert thinks there's a 1% chance we live in a world with a high risk". This changes the appropriate policy implications: if taken at face value they both suggest there's something important to do, but in the latter case the action is less like "work out what to do about simulation shutdown" and more like "start a serious research programme to work out how likely it is we're in the fraction of worlds that was offhand labelled as 1%".

(I also think that you may be right that the cosmologist didn't really believe their numbers. I think that issue alone would be enough reason to be suspicious of the journalist's approach. They might reasonably suspect that such off-the-cuff answers might not withstand scrutiny, so it seems bad to report them in a way that could sound like considered expert opinion.)

I think maybe yes? But I'm a bit worried that "won't react to them" is actually doing a lot of work.

We could chat about more a concrete example that you think fits this description, if you like.

Load more