Lizka's Quick takes

Benjamin Lay — "Quaker Comet", early (radical) abolitionist, general "moral weirdo" — died on this day 267 years ago. 

I shared a post about him a little while back, and still think of February 8 as "Benjamin Lay Day". 

...

Around the same time I also made two paintings inspired by his life/work, which I figured I'd share now. One is an icon-style-inspired image based on a portrait of him^[1]:

The second is based on a print depicting the floor plan of an infamous slave ship (Brooks). The print was used by abolitionists (mainly(?) the Society for Effecting the Abolition of the Slave Trade) to help communicate the horror of the trade. 

I found it useful to paint it (and appreciate having it around today). But I imagine that not everyone might want to see it, so I'll skip a few lines here in case you expanded this quick take and decide you want to scroll past/collapse it instead.

.

.

.

1. ^{^}

   The original (see post) was commissioned by Deborah Read as a gift for her husband Benjamin Franklin, who also Benjamin Lay’s friend.

I've found the following abstract frame/set of heuristics useful for thinking about how we can try to affect (or predict) the long-term future: 

“How do we want to spend our precision/reach points? And can we spend them more wisely?”

[Meta: This is a rough, abstract, and pretty rambly note with assorted links; I’m just trying to pull some stuff out and synthesize it in a way I can more easily reference later (hoping to train habits along these lines). I don't think the ideas here are novel, and honestly I'm not sure who'd find this useful/interesting. (I might also keep editing it as I go.]

An underlying POV here is that (a) scope and (b) precision are in tension. (Alts: (a) "ambition / breadth / reach / ...” — vs —  (b) “predictability / fidelity / robustness / ...”). You can aim at something specific and nearby [high precision, limited reach] or at something larger and farther away, fuzzier [low precision, broad reach]. And if you care about the kind of effect you’re having (you want to make X happen, not just looking for influence ~for influence’s sake), this matters a bunch.

Importantly, I think there are “architectural” features of the world/reality^[1] that can ease this tension somewhat if they're used properly; if you channel your effort through them, you can transmit an intervention without it dissipating (or getting warped) as much as it otherwise would. Any channels like this will still be leaky (and they’re limited), but this sort of “structure” seems like the main thing to look for if you’re hoping to think about or improve the long-term future. 

(See a related sketch diagram here. I also often picture something like: “what levers could reach across a paradigm shift?” (or: what features are invariant in relevant ways?))

Some examples / thinking this through a bit:

1. Trying to organize or steer a social movement (/big group of people) might extend your reach, but you generally sacrifice precision/fidelity by doing that; the group might get derailed, the ideas might mutate (or just get blurrier / diluted), etc.^[2]
   1. Sometimes people build unusually stable social structures (i.e. ones that don’t change that much).^[3] Maybe some religious orders are like this. I think these often exploit a pattern like “encouraging strong+loud commitment to various kinds of norms is self-reinforcing”.
   2. (The broader pattern here might be about creating a thing that can maintain homeostasis, and the trick is finding healthy versions/.)
   3. (On that note, questions like “Where do (did?) stable, cooperative institutions come from?” seem really useful to explore.)
2. Betting on long-lasting institutions could let you extend your reach by leveraging the likely persistent role(?)/relative stability of those institutions
   1. ...but you’re still paying a scope and/or precision/fidelity cost (there are only particular ways that you can shape an institution’s behavior, in some scenarios an institution that looks relevant won’t matter at all for what you care about, it might just get in the way, etc.)
   2. E.g. trying improve a government body’s decision-making with the hope that it’ll help with AI governance
3. Relatedly, particular high-plasticity moments (or catalyzing/disruptive events) might matter a lot, and we might be able to predict some of them+what matters, and be better prepared for them.
   1. E.g. maybe a bunch of countries/actors will come together and write something like a new declaration of human rights. Or something may act as a “warning shot”, and for some (brief?) time high-inertia things like what the public cares about or the structure of a major institution might become very malleable.
   2. This is significant for how I think about AI. (See “the crucible” or “AI as a constitutional moment.”^[4])
   3. The other side of high-plasticity moments is the possibility of crystallization/lock-ins/strong basins. (See dynamism, value lock-in, ...)
4. Other high-level asymmetries can also function as “structure” in this way:
   1. E.g. perhaps truth or certain virtues are self-reinforcing and generally beneficial, or tend to good equilibria; we can use this kind of thing to ~bootstrap
      1. See “Good governance escape velocity”, “promoting x x-ingly”, aligning to virtues... (stuff like superrationality / meta cooperative principles?)
   2. Or a different pattern/feature: if we’re in some not-too-stable social equilibrium (e.g. many people afraid of expressing a view), then acting (as an individual) in a true-to-yourself way can break that equilibrium — see “being a fixed point”.
      1. (A more zoomed out “pattern” here might just be that “good things are good”)

And so the point is that some projects find significantly “smarter” paths through this <reach (things have a small effect) vs precision (things don't have the effect you want)> space, piggybacking on features of reality that are more stable and predictably causally linked. I.e. it helps to orient to ~casual chokepoints that are close enough to predict/act on (we know stuff about them, we can use them as operational targets -- they’re on the right horizon/within reach), but causally upstream of enough important stuff for improvements to propagate down and make a big (positive!) difference. 

I wrote most of this when we were working on the “First type of transformative AI?” post (here's the Forum version); I’d found it very natural to translate some of that into the above frame. 

Something like:

-> People often zero in on something like the "boss battle" AI challenge (e.g. ~ASI), but — even assuming that's the main threat — aiming directly for dealing with later-stage AI transformations seems like it's often an inefficient way of spending your precision/reach points (relative to channeling your effort through shaping earlier AI impacts). 

I.e. (barring something like the "silent-IE" trajectory here) —

If AI will have transformed a bunch of other stuff by the time the issue shows up, the world you're preparing for will be radically different than the one you're used to (and harder to predict). 

• If you try to do something specific/robust/..., there's a greater chance that your work will be irrelevant (or it'll only help with the small slice of things/scenarios you can predict more specifically); your scope is very narrow.
• If you do something much more ambitious, you really need to hope that the intervention is actually helpful instead of getting warped / distracted along the way (moves like trying to start a movement, which could end up harmful or pushing for things that aren't that useful, or e.g. trying to lock in a particular power structure or set of norms before knowing more about what's going on).

(This is basically just the usual pattern. You can maybe think of it as having "how much AI has changed the world" on the X-axis instead of "time into the future")

Meanwhile, earlier-in-the-queue AI impacts might provide us with really good “channels”/levers (for affecting the long-term):

• They're "within reach" — we know more about how AI is changing things now, have more access/ability to change things (plus this is getting less attention)
• In some cases I think shaping how this plays out could be a reasonably good way to faithfully “transmit” the effects of our effort
  • E.g. because we have reason to believe that they’re causally connected to later shifts in particular ways (e.g. the good kind of positive feedback loop on epistemics/coordination; if this change goes better, we’ll be nontrivially better set up for dealing with AI takeover threats / power concentration issues, ...)
  • (For epistemics/coordination effects I also think they'd help quite a bit with other potential challenges -- it's a bit like "general capacity for sense" in my mind. But I expect this "causal connection" step is really tricky more generally and maybe voids some of this frame? )
• I think we can point to some that matter a lot; orienting to them doesn't limit the scope too much (even if they’re not “the big one”);
  • (Even if you only consider the effects on AI-disempowerment threats, I think shaping how e.g. AI gets deployed in information ecosystems could matter a lot.)

To be clear: “how earlier AI impacts go” definitely isn't the only kind of channel/ causal chokepoint we can use to try to improve “how AI goes” or to help with "boss battles" etc. E.g.

• The nature of advanced AI systems is obviously critical and seems (somewhat) predictably affectable; on some views shaping something like ASI is “within reach” (i.e. you should just directly work on something like prosaic superhuman-AI-alignment), or maybe you should work on bootstrapping to safely automated alignment (/virtuous cycles on this front), or maybe the internal processes of the major AI companies are the key thing, and so on.
• I also think the order of different AI capabilities could matter a lot, and we might have some “differential AI development” opportunities.
• ...

Still, I continue to think that people focus too much on something like the "silent-IE" path here, and too little on “in which ways could AI massively change things early — before disempowerment-threatening-ASI — and can we improve how that goes?”

(At the time I also wrote the following (still true): Having written that, I find myself wanting to look closer at the “architecture” of potential early transformations; what are important early transformations that give us some predictability channel? Etc.)

Anyway, I like this as a “find a route to impact/predictability that makes use of more robust/persistent features” prompt. In my mind it’s also quite related to how I would prefer people orient to speculative BOTECs; find “features” related to the question you care about that are more stable (including inputs that are more grounded and a way to put them together that you trust). (Related section of my “ITN 201” post. See also this note on using simpler models.) 

I think similar dynamics apply in a bunch of other places, too.^[5] 

== Some only-very-marginally-relevant images == 

An old sketch illustrating a sort of related idea (trying to see further-out "beacons" is useful, but it's also useful to have a "nearer" target that you can actually visualize, even if it's flawed, which was a major motivation for the design sketches work):

Another rough diagram (from over a year ago, now, I think): 

And just for fun I'll add a couple other sketches in this footnote (people told me around when I made them that they were largely incomprehensible, IIRC).^[6]

1. ^{^}

   Or: levers, causal chokepoints, something about slower-moving layers / something-like-invariants, patterns/ structures, basins/pockets/equilibria/attractors, ...

   "Levers" is the natural word for a bunch of this, but often doesn't quite fit right for me?

2. ^{^}

   People often have quite a bit of choice in how to navigate the ~precision-breadth tradeoff, though, I think. See e.g. The fidelity model of spreading ideas 

3. ^{^}

   It's probably more accurate to say "...sometimes unusually stable social structures arise", actually. 

   My model is that they're often not really designed by anyone.

   Similarly I think “leaders” of broad social structures (including e.g. revolutions) are often "selected in" or "leading the parade" rather than doing something like directing. (A check with Claude suggests that original main leaders/planners of successful revolutions usually did not hold power for at least 5 years [5-7/29 cases, apparently], and ~rarely achieved their stated political goals.)

   Semi-related: (social) system dynamics are quite counterintuitive, “places to intervene in a system.”

^{^}

I also remember appreciating Acemoglu’s Institutions, Technology and Prosperity

^{^}

Related shortform by my brother (bold mine):

[...] This doesn't mean we need to give up, or only work on unambitious, practical applications. But it does mean that we have to admit that things can be useful to work on in expectation before we have a "complete story for how they save the world".

Note that what is being advocated here is not an "anything goes" mentality. I certainly think that AI safety research can be too abstract, too removed from any realistic application in any world. But there is a large spectrum of possibilities between "fully plan how you will solve a complex logic game before trying anything" and "make random jerky moves because they 'feel right'".

I'm writing this in response to Adam Jones' article on AI safety content.. I like a lot of the suggestions. But I think the section on alignment plans suffers from the "axe" fallacy that I claim is somewhat endemic here. Here's the relevant quote:

> For the last few weeks, I’ve been working on trying to find plans for AI safety. They should cover the whole problem, including the major hurdles after intent alignment. Unfortunately, this has not gone well - my rough conclusion is that there aren’t any very clear and well publicised plans (or even very plausible stories) for making this go well. (More context on some of this work can be found in BlueDot Impact’s AI safety strategist job posting). (emphasis mine).

I strongly disagree with this being a good thing to do!

We're not going to have a good, end-to-end plan about how to save the world from AGI. Even now, with ever more impressive and scary AIs becoming a comonplace, we have very little idea about what AGI will look like, what kinds of misalignment it will have, where the hard bits of checking it for intent and value alignment will be. Trying to make extensive end-to-end plans can be useful, but can also lead to a strong streetlight effect: we'll be overcommitting to current understanding, current frames of thought (in an alignment community that is growing and integrating new ideas with an exponential rate that can be factored in months, not years).

Don't get me wrong. I think it's valuable to try to plan things where our current understanding is likely to at least partially persist: how AI will interface with government, general questions of scaling and rough models of future development. But we should also understand that our map has lots of blanks, especially when we get down to thinking about what we will understand in the future.  [...]

A few other links I'd dumped in a doc with this note: 

John Wentworth’s writing on gears-level models (e.g. “...are capital investments”). 

Eliezer:

 - The Outside View's Domain ("...does not inspire in me any confidence that the Outside View can be applied across processes with greatly different internal causal structures, like life-and-death versus sleeping-and-waking. ..." ... "when you deal with attempted analogies across structually different processes, perhaps unique or poorly understood, then things which are similar in some surface respects are often different in other respects.  And the sign of this domain is that when people try to reason by similarity, it is not at all clear what is similar to what, or which surface resemblances they should focus upon as opposed to others.") 

 - Underconstrained Abstractions ("...The further away you get from highly regular things like atoms, and the closer you get to surface phenomena that are the final products of many moving parts, the more history underconstrains the abstractions that you use.  This is part of what makes futurism difficult. ")

(also this sequence)

^{^}

From "First type of TAI?" again, a whacky schematic:



Whacky illustration: 



Show all footnotes

When I try to think about how much better the world could be, it helps me to sometimes pay attention to the less obvious ways that my life is (much) better than it would have been, had I been born in basically any other time (even if I was born among the elite!).

So I wanted to make a quick list of some “inconspicuous miracles” of my world. This isn’t meant to be remotely exhaustive, and is just what I thought of as I was writing this up. The order is arbitrary.

1. Washing machines

It’s amazing that I can just put dirty clothing (or dishes, etc.) into a machine that handles most of the work for me. (I’ve never had to regularly hand-wash clothing, but from what I can tell, it is physically very hard, and took a lot of time. I have sometimes not had a dishwasher, and really notice the lack whenever I do; my back tends to start hurting pretty quickly when I’m handwashing dishes, and it’s easier to start getting annoyed at people who share the space.) 

Source: Illustrations from a Teenage Holiday Journal (1821). See also.

2. Music (and other media!)

Just by putting on headphones, I get better music performances than most royalty could ever hope to see in their courts. (The first recording was made in the 19th century. Before that — and also widespread access to radios — few people would get access to performances by “professional” musicians, maybe besides church or other gatherings. (Although I kind of expect many people were somewhat better at singing than most are today.)) (A somewhat silly intuition pump: apparently Franz Liszt inspired a mania. But most 19th century concert attendees wouldn’t be hearing him more than once or maybe a handful of times! So imagine listening to a piece that rocks your world, and then only being able to remember it by others’ recreations or memorabilia you’ve managed to get your hands on.) 

In general, entertainment seems so much better today. Most people in history were illiterate — their entertainment might come from their own experiences or maybe telling each other stories, games, and occasional plays or other events. Of the people who could, at least, read books, most had access to very few. Today, we have a huge number of books and other texts, as well as TV/movies, and more. I have access to video demonstrations of a functionally endless number of things, (color!) photos of an enormous amount of art (I remember being struck by my mom mentioning that she’d always wished to see some building or other when she was younger, as she’d seen it in a book that had a catalogue of black-and-white architecture pictures at some point), etc.

People listening to a touring phonograph, from the 1900 inaugural issue of Boletín fonográfico. Source: Inventing the recording.

3. Glasses

I have pretty mild shortsightedness (with a mild astigmatism) but even then, I’m really glad that I have glasses; wearing them prevents headaches/ fatigue, makes it easier for me to see people’s faces (or street signs), and makes a huge difference if I’m trying to see a whiteboard (or watch a show or movie). And a number of my friends and family are near-blind without glasses, in ways that make me feel a lot of sympathy for people in the past; eye problems aren’t new. (In some ways, the modern world requires us to engage more with things like written signage, but I think many people in history had duller or more painful lives because they lacked glasses.)

Source: Highlights from Folger Shakespeare Library’s Release of almost 80,000 Images

4. Not having to marry someone against my will

By the standards of most of history, I’d be something like an old maid. I’m so grateful that I’ve never felt pressure of any kind to marry someone I didn’t want to. 

Source.

5. Waterproof shoes / umbrellas/ coats

I live in Oxford, and have shoes in which I can walk through ankle-high puddles without wetting my feet. But whenever I get my feet wet and have to stay out in the cold — if I forget to bring an umbrella or coat and it rains heavily — I get quite miserable! Rain isn’t a modern development; a lot of people had to deal with these problems a lot. Moreover, most people’s houses were leakier and significantly less insulated than mine. 

Source: Wikimedia

6. Photos and videos of loved ones, ability to (video) call them 

We have very few pictures of my brother when he was very young (before my parents moved to the US), and very few childhood photos of my parents or older relatives. There are significantly more pictures of me, and I took tons of photos of my little sister — and love looking at them. And I can call my family and friends when we’re not in the same place! I don’t need to wait weeks to send and receive letters, hoping that they will arrive, or rely on sketches or miniatures. 

7. Sewage systems, garbage disposal, menstrual products, diapers, etc.

I don’t need to use an outhouse, and walk along streets that are basically free of sewage and trash. I’m not chronically ill; I can be pretty confident that the water I drink isn’t contaminated by excrement. My life is basically normal when I’m on my period. When I’ve changed babies, I could just throw away the diaper.

Source.

I don’t want to pretend that life in earlier periods was all bad! (People had solutions to some of these problems that meant they could handle them, in some cases, better than I can today when they arose. And people had wonderful and meaningful moments — here’s a really touching collection of photos of friendship I saw as I was finding the pictures above.) But I’m so grateful for some things that have changed at least in some places. 

What do I want to see in the future?

The world could be so much better. There are daily tragedies or horrors that need to stop, risks we face that I want to reduce, and privileges enjoyed by some that I want to see extended to everyone else. But even the ~daily standards of today’s privileged groups could go up by a lot. For instance, I think our treatment and practices around mental health / psychological pain are still far from what they could be — I often feel like we’re in the 18th century equivalent of medicine on this front. Chronic pain management and aging science (e.g handling dementia) is also significantly farther behind what it could be. Pain management (and healthcare more generally) has improved massively over the last decades/centuries, but people still have horrible migraines, long diseases, etc. I think we still have less quality time to spend with people we love and on things we find meaningful than we could. Technology could augment (or reduce the need for) our willpower in some areas, etc. 

Source.

When thinking about the impacts of AI, I’ve found it useful to distinguish between different reasons for why automation in some area might be slow. In brief: 

1. raw performance issues
2. trust bottlenecks
3. intrinsic premiums for “the human factor”
4. adoption lag
5. motivated/active protectionism towards humans

I’m posting this mainly because I’ve wanted to link to this a few times now when discussing questions like "how should we update on the shape of AI diffusion based on...?". Not sure how helpful it will be on its own!

In a bit more detail:

(1) Raw performance issues

There’s a task that I want an AI system to do. An AI system might be able to do it in the future, but the ones we have today just can’t do it. 

For instance:

• AI systems still struggle to stay coherent over long contexts, so it’s often hard to use an AI system to build out a massive codebase without human help, or write a very consistent detective novel.
• Or I want an employee who’s more independent; we can get aligned on some goals and they will push them forward, coming up with novel pathways, etc.
• (Other capability gaps: creativity/novelty, need for complicated physical labor, …)
  • [Edit: this UK AISI paper on limitations of current AI looks good]

A subclass here might be performance issues that are downstream of “interface mismatch”.^[1] Cases where AI might be good enough at some fundamental task that we’re thinking of (e.g. summarizing content, or similar), but where the systems that surround that task or the interface through which we’re running the thing — which are trivial for humans — is a very poor fit for existing AI, and AI systems struggle to get around that.^[2] (E.g. if the core concept is presented via a diagram, or requires computer use stuff.) In some other cases, we might separately consider whether the AI systems has the right affordances at all. 

This is what we often think about when we think about the AI tech tree / AI capabilities. Others are often important, though: 

(2) Verification & trust bottlenecks

The AI system might be able to do the task, but I can't easily check its work and prefer to just do it myself or rely on someone I trust. Or I can't be confident the AI won't fail spectacularly in some rare but important edge cases (in ways a human almost certainly wouldn't).^[3]

For instance, maybe I want someone to pull out the most surprising and important bits from some data, and I can’t trust the judgement of an AI system the way I can trust someone I’ve worked with. Or I don’t want to use a chatbot in customer-facing stuff in case someone finds a way to make it go haywire.

A subclass here is when one can’t trust AI providers (and open-source/on-device models aren’t good enough) for some use case. Accountability sinks also play a role on the “trust” front. Using AI for some task might complicate the question of who bears responsibility when things go wrong, which might matter if accountability is load-bearing for that system. In this case we might go for assigning a human overseer.^[4]

(3) Intrinsic premiums for “the human factor”^[5]

The task *requires* human involvement, or I intrinsically prefer it for some reason.

E.g. AI therapists are less effective because knowing that the person on the other end is a person is actually useful to get me to do my exercises. Or: I might pay more to see a performance by Yo Yo Ma than by a robot because that’s my true preference; I get less value from a robot performance.

A subclass here is cases where the value I’m getting is testimony — e.g. if I want to gather user data or understand someone’s internal experience — that only humans can provide (even if AI can superficially simulate it).

(4) Adoption lag & institutional inertia

AI systems can do this, people would generally prefer that, there aren’t legal or similar “hard barriers” to AI use, etc., but in practice this is being done by humans. 

E.g. maybe AI-powered medical research isn’t happening because the people/institutions who could be setting this kind of automation up just haven’t gotten to it yet. 

Adoption lags might be caused by stuff like: sheer laziness, coordination costs, lack of awareness/expertise, attention scarcity — the humans currently involved in this process don’t have enough slack — or similar, active (but potentially hidden) incumbent resistance, or maybe bureaucratic dysfunction (no one has the right affordances).  

(My piece on the adoption gap between the US government and ~industry is largely about this.)

(5) Motivated/active protectionism towards humans

AI systems that can do this, there’s no intrinsic need for human involvement, nor real capabilities/trust bottlenecks getting in the way. But we’ll deliberately continue relying on human labor for ~political reasons — not just because we’re moving slowly. 

E.g. maybe we’ve hard-coded a requirement that a human lawyer or teacher or driver (etc.) is involved. A particularly salient subclass/set of examples here is when a group of humans has successfully lobbied a government to require human labor (where it might be blatantly obvious that it’s not needed). In other cases, the law (or anatomy of an institution) might incidentally require humans in the loop via some other requirement.

This is a low-res breakdown that might be missing stuff. And the lines between these categories can be very fuzzy. For instance, verification difficulties (2) can provide justification for protectionism (5) or further slow down adoption (4). 

But I still think it’s useful to pay attention to what’s really at play, and worry we too often think exclusively in terms of raw performance issues (1) with a bit of diffusion lag (4). 

1. ^{^}

   Note OTOH that sometimes the AI capabilities are already there, but bad UI or lack of some complementary tech is still making those capabilities unusable.

2. ^{^}

   I’m listing this as a “raw performance issue” because the AI systems/tools will probably keep improving to better deal with such clashes. But I also expect the surrounding interfaces/systems to change as people try to get more value from available AI capabilities. (E.g. stuff like robots.txt.)

3. ^{^}

   Sometimes the edge cases are the whole point. See also: Heuristics That Almost Always Work - by Scott Alexander 

^{^}

Although I guess then we should be careful about alert fatigue/false security issues. IIRC this episode discusses related stuff: Machine learning meets malware, with Caleb Fenton (search for “alert fatigue” / “hypnosis”) 

^{^}

(There's also the opposite; cases where we'd actively prefer humans not be involved. For instance, all else equal, I might want to keep some information private — not want to share it with another person — even if I'd happily share with an AI system.)

Show all footnotes

Notes on some of my AI-related confusions^[1]

It’s hard for me to get a sense for stuff like “how quickly are we moving towards the kind of AI that I’m really worried about?” I think this stems partly from (1) a conflation of different types of “crazy powerful AI”, and (2) the way that benchmarks and other measures of “AI progress” de-couple from actual progress towards the relevant things. Trying to represent these things graphically helps me orient/think. 

First, it seems useful to distinguish the breadth or generality of state-of-the-art AI models and how able they are on some relevant capabilities. Once I separate these out, I can plot roughly where some definitions of "crazy powerful AI" apparently lie on these axes: 

(I think there are too many definitions of "AGI" at this point. Many people would make that area much narrower, but possibly in different ways.)

Visualizing things this way also makes it easier for me^[2] to ask: Where do various threat models kick in? Where do we get “transformative” effects? (Where does “TAI” lie?)

Another question that I keep thinking about is something like: “what are key narrow (sets of) capabilities such that the risks from models grow ~linearly as they improve on those capabilities?” Or maybe “What is the narrowest set of capabilities for which we capture basically all the relevant info by turning the axes above into something like ‘average ability on that set’ and ‘coverage of those abilities’, and then plotting how risk changes as we move the frontier?”

The most plausible sets of abilities like this might be something like: 

• Everything necessary for AI R&D^[3]
• Long-horizon planning and technical skills?

If I try the former, how does risk from different AI systems change? 

And we could try drawing some curves that represent our  guesses about how the risk changes as we make progress on a narrow set of AI capabilities on the x-axis. This is very hard; I worry that companies focus on benchmarks in ways that make them less meaningful, so I don’t want to put performance on a specific benchmark on the x-axis. But we could try placing some fuzzier “true” milestones along the way, asking what the shape of the curve would be in reference to those, and then trying to approximate how far along we are with respect to them by using a combination of metrics and other measures. (Of course, it’s also really difficult to develop a reasonable/useful sense for how far apart those milestones are on the most appropriate measure of progress — or how close partial completion of these milestones is to full completion.)

Here's a sketch:

Overall I'm really unsure of which milestones I should pay attention to here, and how risk changes as we might move through them.

It could make sense to pay attention to real-world impacts of (future) AI systems instead of their ~intrinsic qualities, but real-world impacts seem harder to find robust precursors to, rely on many non-AI factors, and interpreting them involves trying to untangle many different cruxes or worldviews. (Paying attention to both intrinsic qualities and real-world impacts seems useful and important, though.)

All of this also complicates how I relate to questions like “Is AI progress speeding up or slowing down?” (If I ignore all of these confusions and just try to evaluate progress intuitively / holistically, it doesn't seem to be slowing down in relevant ways.^[4])

Thoughts/suggestions/comments on any of this are very welcome (although I may not respond, at least not quickly).

Some content related to at least some of the above (non-exhaustive): 

• METR’s RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts
• AI Impacts page on HLAI, especially “Human-level” is superhuman
• List of some definitions of advanced AI systems
• John Wentworth distinguishing between “early transformative AI” and “superintelligence”
• Holden Karnofsky's recent piece for the Carnegie Endowment for International Peace: AI Has Been Surprising for Years
• Writing on (issues with) benchmarks/evals: Kelsey Piper in Vox, Anthropic's "challenges in evaluating AI systems", Epoch in 2023 on how well compute predicts benchmark performance (pretty well on average, harder individually)
• Recent post that I appreciated, which outlined timelines via some milestones and then outlined a picture of how the world might change in the background

1. ^{^}

   Not representing "Forethought views" here! (I don't know what Forethought folks think of all of this.)

   Written/drawn very quickly. 

2. ^{^}

   This diagram also makes me wonder how much pushing the bottom right corner further to the right (on some relevant capabilities) would help (especially as an alternative to pushing up or diagonally), given that sub-human general models don't seem that safety-favoring but some narrow-but-relevant superhuman capabilities could help us deal with the risks of more general, human-level+ systems.

3. ^{^}

   Could a narrower set work? E.g. to what extent do we just care about ML engineering? 

^{^}

although I'm somewhat surprised at the lack of apparent real-world effects

Show all footnotes

USAID has announced that they've committed $4 million to fighting global lead poisoning! 

USAID Administrator Samantha Power also called other donors to action, and announced that USAID will be the first bilateral donor agency to join the Global Alliance to Eliminate Lead Paint. The Center for Global Development (CGD) discusses the implications of the announcement here. 

For context, lead poisoning seems to get ~$11-15 million per year right now, and has a huge toll. I'm really excited about this news.

Also, thanks to @ryancbriggs for pointing out that this seems like "a huge win for risky policy change global health effective altruism" and referencing this grant:

In December 2021, GiveWell (or the EA Funds Global Health and Development Fund?) gave a grant to CGD to "to support research into the effects of lead exposure on economic and educational outcomes, and run a working group that will author policy outreach documents and engage with global policymakers." In their writeup, they recorded a 10% "best case" forecast that in two years (by the end of the grant period), "The U.S. government, other international actors (e.g., bilateral and multilateral donors), and/or national LMIC governments take measurable action to reduce lead exposure—for example, through increased funding for lead mitigation and research, increased monitoring of lead exposure, and/or enactment of regulations." We've reached this best case and it's been almost exactly two years! (Attributing credit is really hard and I have no experience and little context in this area — as far as I know this could have happened without that grant or related advocacy. But it's still notable to me that a CGD report is cited in Power's announcement.)

A note on how I think about criticism

(This was initially meant as part of this post,^[1] but while editing I thought it didn't make a lot of sense there, so I pulled it out.)

I came to CEA with a very pro-criticism attitude. My experience there reinforced those views in some ways,^[2] but it also left me more attuned to the costs of criticism (or of some pro-criticism attitudes). (For instance, I used to see engaging with all criticism as virtuous, and have changed my mind on that.) My overall takes now aren’t very crisp or easily summarizable, but I figured I'd try to share some notes.

...

It’s generally good for a community’s culture to encourage criticism, but this is more complicated than I used to think.

Here’s a list of things that I believe about criticism:

1. Criticism or critical information can be extremely valuable. It can be hard for people to surface criticism (e.g. because they fear repercussions), which means criticism tends to be undersupplied.^[3] Requiring critics to present their criticisms in specific ways will likely stifle at least some valuable criticism. It can be hard to get yourself to engage with criticism of your work or things you care about. It’s easy to dismiss true and important criticism without noticing that you’re doing it. 
   1. → Making sure that your community’s culture appreciates criticism (and earnest engagement with it), tries to avoid dismissing critical content based on stylistic or other non-fundamental qualities, encourages people to engage with it, and disincentivizes attempts to suppress it can be a good way to counteract these issues. 
2. At the same time, trying to actually do anything is really hard.^[4] Appreciation for doers is often undersupplied. Being in leadership positions or engaging in public discussions is a valuable service, but opens you up to a lot of (often stressful) criticism, which acts as a disincentive for being public. Psychological safety is important in teams (and communities), so it’s unfortunate that critical environments lead more people to feel like they would be judged harshly for potential mistakes. Not all criticism is useful enough to be worth engaging with (or sharing). Responding to criticism can be time-consuming or otherwise costly and isn’t always worth it.^[5] Sometimes people who are sharing “criticism” hate the project for reasons that aren’t what’s explicitly stated, or just want to vent or build themselves up.^[6]
   1. ... and cultures like the one described above can exacerbate these issues.

I don’t have strong overall recommendations. Here’s a post on how I want to handle criticism, which I think is still accurate. I also (tentatively) think that on the margin, the average person in EA who is sharing criticism of someone’s work should probably spend a bit more time trying to make that criticism productive. And I’d be excited to see more celebration or appreciation for people’s work. (I also discussed related topics in this short EAG talk last year.)

1. ^{^}

   This was in that post because I ended up engaging with a lot of discussion about the effects of criticism in EA (and of the EA Forum’s critical culture) as part of running a Criticism Contest (and generally working on CEA’s Online Team).

2. ^{^}

   I’ve experienced first-hand how hard it is to identify flaws in projects you’re invested in, I’ve seen how hard it is for some people to surface critical information, and noticed some ways in which criticism can be shut down or disregarded by well-meaning people.

3. ^{^}

   See also the rationale in our Criticism Contest announcement post.

^{^}

Kinda related: EA should taboo "EA should" 

^{^}

 See the “transparency” example in my post on “missing moods”. 

Also: You don’t have to respond to every comment.

^{^}

A lot of what Richard says in Moral Misdirection (and in Anti-Philanthropic Misdirection) also seems true and relevant here.

Show all footnotes

A note on mistakes and how we relate to them

(This was initially meant as part of this post^[1], but I thought it didn't make a lot of sense there, so I pulled it out.)

“Slow-rolling mistakes” are usually much more important to identify than “point-in-time blunders,”^[2] but the latter tend to be more obvious.

When we think about “mistakes”, we usually imagine replying-all when we meant to reply only to the sender, using the wrong input in an analysis, including broken hyperlinks in a piece of media, missing a deadline, etc. I tend to feel pretty horrible when I notice that I've made a mistake like this.

I now think that basically none of my mistakes of this kind — I’ll call them “Point-in-time blunders” — mattered nearly as much as other "mistakes" I've made by doing things like planning my time poorly, delaying for too long on something, setting up poor systems, or focusing on the wrong things. 

This second kind of mistake — let’s use the phrase “slow-rolling mistakes” — is harder to catch; I think sometimes I'd identify them by noticing a nagging worry, or by having multiple conversations with someone who disagreed with me (and slowly changing my mind), or by seriously reflecting on my work or on feedback I'd received. 

...

This is not a novel insight, but I think it was an important thing for me to realize. Working at CEA helped move me in this direction. A big factor in this, I think, was the support and reassurance I got from people I worked with. 

This was over two years ago, but I still remember my stomach dropping when I realized that instead of using “EA Forum Digest #84” as the subject line for the 84th Digest, I had used “...#85.” Then I did it AGAIN a few weeks later (instead of #89). I’ve screenshotted Ben’s (my manager’s) reaction.

...

I discussed some related topics in a short EAG talk I gave last year, and also touched on these topics in my post about “invisible impact loss”. 

An image from that talk.

1. ^{^}

   It was there because my role gave me the opportunity to actually notice a lot of the mistakes I was making (something that I think is harder if you’re working on something like research, or in a less public role), which also meant I could reflect on them. 

2. ^{^}

   If you have better terms for these, I'd love suggestions!

Reflection on my time as a Visiting Fellow at Rethink Priorities this summer

I was a Visiting Fellow at Rethink Priorities this summer. They’re hiring right now, and I have lots of thoughts on my time there, so I figured that I’d share some. I had some misconceptions coming in, and I think I would have benefited from a post like this, so I’m guessing other people might, too. Unfortunately, I don’t have time to write anything in depth for now, so a shortform will have to do.

Fair warning: this shortform is quite personal and one-sided. In particular, when I tried to think of downsides to highlight to make this post fair, few came to mind, so the post is very upsides-heavy. (Linch’s recent post has a lot more on possible negatives about working at RP.) Another disclaimer: I changed in various ways during the summer, including in terms of my preferences and priorities. I think this is good, but there’s also a good chance of some bias (I’m happy with how working at RP went because working at RP transformed me into the kind of person who’s happy with that sort of work, etc.). (See additional disclaimer at the bottom.)

First, some vague background on me, in case it’s relevant:

• I finished my BA this May with a double major in mathematics and comparative literature.
• I had done some undergraduate math research, had taught in a variety of contexts, and had worked at Canada/USA Mathcamp, but did not have a lot of proper non-Academia work experience.
• I was introduced to EA in 2019.

Working at RP was not what I had expected (it seems likely that my expectations were skewed).

One example of this was how my supervisor (Linch) held me accountable. Accountability existed in such a way that helped me focus on goals (“milestones”) rather than making me feel guilty about falling behind. (Perhaps I had read too much about bad workplaces and poor incentive structures, but I was quite surprised and extremely happy about this fact.) This was a really helpful transition for me from the university context, where I often had to complete large projects with less built-in support. For instance, I would have big papers due as midterms (or final exams that accounted for 40% of a course grade), and I would often procrastinate on these because they were big, hard to break down, and potentially unpleasant to work on. (I got really good at writing a 15-page draft overnight.)

In contrast, at Rethink, Linch would help me break down a project into steps (“do 3 hours of reading on X subject,” “reach out to X person,” “write a rough draft of brainstormed ideas in a long list and share it for feedback,” etc.), and we would set deadlines for those. Accomplishing each milestone felt really good, and kept me motivated to continue with the project. If I was behind the schedule, he would help me reprioritize and think through the bottlenecks, and I would move forward. (Unless I’m mistaken, managers at RP had taken a management course in order to make sure that these structures worked well — I don’t know how much that helped because I can’t guess at the counterfactual, but from my point of view, they did seem quite prepared to manage us.)

Another surprise: Rethink actively helped me meet many (really cool) people (both when they did things like give feedback, and through socials or 1-1’s). I went from ~10 university EA friends to ~25 people I knew I could go to for resources or help. I had not done much EA-related work before the internship (e.g. my first EA Forum post was due to RP), but I never felt judged or less respected for that. Everyone I interacted with seemed genuinely invested in helping me grow. They sent me relevant links, introduced me to cool new people, and celebrated my successes.

I also learned a lot and developed entirely new interests. My supervisor was Linch, so it might be unsurprising that I became quite interested in forecasting and related topics. Beyond this, however, I found the work really exciting, and explored a variety of topics. I read a bunch of economics papers and discovered that the field was actually really interesting (this might not be a surprise to others, but it was to me!). I also got to fine-tune my understanding of and opinions on a number of questions in EA and longtermism. I developed better work (or research) habits, gained some confidence, and began to understand myself better.

Here’s what I come up with when I try to think of negatives:

• I struggled to some extent with the virtual setting (e.g. due to tech or internet issues). Protip: if you find yourself with a slow computer, fix that situation asap.
• There might have been too much freedom for me— I probably spent too long choosing and narrowing my next project topics. Still, this wasn’t purely negative; I think I ended up learning a lot during the exploratory interludes (where I went on deep-dives into things like x-risks from great power conflict, but they did not help me produce outputs). As far as I know, this issue is less relevant for more senior positions, and a number of more concrete projects are more straightforwardly available now. (It also seems likely that I could have mitigated this by realizing it would be an issue right away.)
• I would occasionally fall behind and become stressed about that. A few tasks became ugh fields. As the summer progressed, I think I got better about immediately telling Linch when I noticed myself feeling guilty or unhappy about a project, and this helped a lot.
• Opportunity cost. I don’t know exactly what I would have done during the summer if not RP, but it’s always possible it would have been better.

Obviously, if I were restarting the summer, I would do some things differently. I might focus on producing outputs faster. I might be more active in trying to meet people. I would probably organize my daily routine differently. But some of the things I list here are precisely changes in my preferences or priorities that result from working at RP. :)

I don’t know if anyone will have questions, but feel free to ask questions if you do have any. But I should note that I might not be able to answer many, as I’m quite low on free time (I just started a new job).

Note: nobody pressured me to write this shortform, although Linch & some other people at RP did know I was doing it and were happy for it. For convenience, here’s a link to RP’s hiring page.

TLDR: Notes on confusions about what we should do about digital minds, even if our assessments of their moral relevance are correct^[1]

I often feel quite lost when I try to think about how we can “get digital minds right.” It feels like there’s a variety of major pitfalls involved, whether or not we’re right about the moral relevance of some digital minds.

Categories (2) and (3) above are ultimately about being confused about which digital minds matter morally — having an inappropriate level of concern, one way or another. A lot of the current research on digital minds seems to be aimed at avoiding this issue. (See e.g. and e.g..) I’m really glad to see this work; the pitfalls in these categories worry me a lot. 

But even if we end up in category (1) — we realize correctly that certain digital minds are morally relevant — how can we actually figure out what we should do? Understanding how to respond probably involves answering questions like: 

• What matters for these digital minds? What improves their experience? (How can we tell?)
• Is their experience overall negative/positive? Should we be developing such minds? (How can we tell??) (How can we ethically design digital minds?)
• Should we respect certain rights for these digital minds? (..?)
• ??

Answering these questions seem really difficult. 

In many cases, extrapolating from what we can tell about humans seems inappropriate.^[4] (Do digital minds systems find joy or meaning in some activities? Do they care about survival? What does it mean for a digital mind/AI system to be in pain? Is it ok to lie to AI systems? Is it ok to design AI systems that have no ~goals besides fulfilling requests from humans?)

And even concepts we have for thinking about what matters to humans (or animals) often seem ill-suited for helping us with digital minds.^[5] (When does it make sense to talk about freedoms and rights,^[6] or the sets of (relevant) capabilities a digital mind has, or even their preferences? What even is the right “individual” to consider? Self-reports seem somewhat promising, but when can we actually rely on them as signals about what’s important for digital minds?)

I’m also a bit worried that too much of the work is going towards the question of “which systems/digital minds are morally relevant” vs to the question of “what do we do if we think that a system is morally relevant (or if we’re unsure)?” (Note however that I've read a tiny fraction of this work, and haven't worked in the space myself!) E.g. this paper focuses on the former and closes by recommending that companies prepare to make thoughtful decisions about how to treat the AI systems identified as potentially morally significant by (a) hiring or appointing a DRI (directly responsible individual) for AI welfare and (b) developing certain kinds of frameworks for AI welfare oversight. These steps do seem quite good to me (at least naively), but — as the paper explicitly acknowledges — they’re definitely not sufficient. 

Some of the work that’s apparently focused at the question of "which digital minds matter morally" involves working towards theories of ~consciousness, and maybe that will also help us with the latter question. But I’m not sure. 

(So my quite uninformed independent impression is that it might be worth investing a bit more in trying to figure out what we should do if we do decide that some digital minds might be morally relevant, or maybe what we should do if we find that we’re making extremely little progress on figuring out whether they are.) 

These seem like hard problems/questions, but I want to avoid defeatism. 

I appreciated @rgb‘s closing remark in this post (bold mine): 

To be sure, our neuroscience tools are way less powerful than we would like, and we know far less about the brain than we would like. To be sure, our conceptual frameworks for thinking about sentience seem shaky and open to revision. Even so, trying to actually solve the problem by constructing computational theories which try to explain the full range of phenomena could pay significant dividends. My attitude towards the science of consciousness is similar to Derek Parfit’s attitude towards ethics: since we have only just begun the attempt, we can be optimistic.

Some of the other content I’ve read/skimmed feels like it’s pointing in useful directions on these fronts (and more recommendations are welcome!): 

• Improving the Welfare of AIs: A Nearcasted Proposal - Ryan Greenblatt on LW
• To understand AI sentience, first understand it in animals - Jonathan Birch in Aeon (I’ve only skimmed this, though)
• ​​Jeff Sebo on digital minds, and how to avoid sleepwalking into a major moral catastrophe - 80,000 Hours (I’ve only skimmed this, again) 

1. ^{^}

   I’m continuing my quick take spree, with a big caveat that I’m not a philosopher and haven’t read nearly as much research/writing on digital minds as I want to.

   And I’m not representing Forethought here! I don’t know what Forethought folks think about what I’m writing here.

2. ^{^}

   The more appropriate term, I think, is “moral patienthood.” And we probably care about the degree of moral consideration that is merited in different cases.

3. ^{^}

   This paper on “Sharing the world with digital minds” mentions this as one of the many failure modes — note that I’ve only skimmed it. 

^{^}

 I’m often struck by how appropriate the War with the Newts is as an analogy/illustration/prediction for a bunch of what I’m seeing today, including on issues related to digital minds. But there’s one major way in which the titular newts differ significantly from potential morally relevant digital minds; the newts are quite similar to humans and can, to some extent, communicate their preferences in ways the book’s humans could understand if they were listening. 

(One potential exception: there’s a tragic-but-brilliant moment in which “The League for the Protection of Salamanders” calls women to action sewing modest skirts and aprons for the Newts in order to appease the Newts’ supposed sense of propriety.)

^{^}

This list of propositions touches on related questions/ideas.

^{^}

 See e.g. here: 

Whether instrumentally-justified or independently binding, the rights that some AI systems could be entitled to might be different from the rights that humans are entitled to. This could be because, instrumentally, a different set of rights for AI systems promotes welfare. For example, as noted by Shulman and Bostrom (2021), naively granting both “reproductive” rights and voting rights to AI systems would have foreseeably untenable results for existing democratic systems: if AI systems can copy themselves at will, and every copy gets a vote, then elections could be won via tactical copying. This set of rights would not promote welfare and uphold institutions in the same way that they do for humans. Or AI rights could differ because, independently of instrumental considerations, their different properties entitle them to different rights—analogously to how children and animals are plausibly entitled to different rights than adults.

Show all footnotes

People have pretty different background expectations about what the most relevant or worrying kind of AI misalignment /takeover/... scenario would  look like. This also corresponds to different views on when they expect signs of it to be visible (such that not seeing those signs or seeing something else would update them). Among other issues, I think this confuses discussion around whether (e.g.) "alignment is easy" or how we should be updating.^[1] 

My brain likes pictures, so I've found it useful to tag different views and discussions via the following diagrams (these are pretty "raw"/not-distilled): 

1)

2) And a second one, roughly: 

how much systems at a given capability level appear safe vs where they actually are on the path or spectrum to the kind of safety we care about^[2]. 

(This also has some more notes on how people might relate differently to the same results / evidence.)

These are very messy sketches! I'm sharing because I made a hacky commitment to post short things and in case it's useful for someone (or in case a comment helps clarify things for me, which I'd definitely appreciate). There's some chance that I'll clean these up and update them later.

1. ^{^}

   Related: "conflationary alliances" (see also a post with a version of this dynamic about "charity" on the Forum)

2. ^{^}

   Again, a huge part of the problem/confusion seems to be that this is a very underdetermined term; see footnote above. Also feels sort of related to things I wrote about here

   (altohugh I'm guessing it's also partly due to the fact that this is a basically unedited sketch and I first drew this because a similar image had come to mind in a variety of contexts, and I wanted a version I could adapt as needed - i.e. it was meant to be flexible. If I were making a v2 I'd probably want to commit more, though.)

Edit: I've now shared: Donation Election: how voting will work. Really grateful for the discussion on this thread!

We're planning on running a Donation Election for Giving Season. 

What do you think the final voting mechanism should be, and why? E.g. approval voting, ranked-choice voting, quadratic voting, etc. 

Considerations might include: how well this will allocate funds based on real preferences, how understandable it is to people who are participating in the Donation Election or following it, etc. 

I realize that I might be opening a can of worms, but I'm looking forward to reading any comments! I might not have time to respond.

Some context (see also the post): 

Users will be able to "pre-vote" (to signal that they're likely to vote for some candidates, and possibly to follow posts about some candidates), for as many candidates as they want. The pre-votes are anonymous (as are final votes), but the total numbers will be shown to everyone. There will be a separate process for final voting, which will determine the three winners in the election. The three winners will receive the winnings from the Donation Election Fund, split proportionally based on the votes.

Only users who had an account as of October 22, 2023, will be able to vote, unfortunately. We’ve had to add this restriction to avoid election manipulation (we’ll also be monitoring in other ways). I realize that this limits genuine new users’ ability to vote, but hopefully the fact that newer users can participate in other ways (like by encouraging others to vote for some candidates, or by donating to candidates/the Donation Election Fund) helps a bit. 

A quick preview of the pre-votes, in case you're interested: 

Vasili Arkhipov is discussed less on the EA Forum than Petrov is (see also this thread of less-discussed people). I thought I'd post a quick take describing that incident.

Arkhipov & the submarine B-59’s missile

On October 27, 1962 (during the Cuban Missile Crisis), the Russian diesel-powered submarine B-59 started experiencing^[1] nearby depth charges from US forces above them; the submarine had been detected and US ships seemed to be attacking. The submarine’s air conditioning was broken,^[2] CO2 levels were rising, and B-59 was out of contact with Moscow. Two of the senior officers on the submarine, thinking that a global war had started, wanted to launch their “secret weapon,” a 10-kiloton nuclear torpedo. The captain, Valentin Savistky, apparently exclaimed: “We’re gonna blast them now! We will die, but we will sink them all — we will not become the shame of the fleet.” 

The ship was authorized to launch the torpedo without confirmation from Moscow, but all three senior officers on the ship had to agree.^[3] Chief of staff of the flotilla Vasili Arkhipov refused. He convinced Captain Savitsky that the depth charges were signals for the Soviet submarine to surface (which they were) — if the US ships really wanted to destroy the B-59, they would have done it by now. (Part of the problem seemed to be that the Soviet officers were used to different signals than the ones the Americans were using.) Arkhipov calmed the captain down^[4] and got him to surface the submarine to get orders from the Kremlin, which ended up eventually defusing the situation. 

(Here's a Vox article on the incident.)

The B-59 submarine.

1. ^{^}

   Vadim Orlov described the impact of the depth charges as being inside an oil drum getting struck with a sledgehammer.

2. ^{^}

   Temperatures were apparently above 45ºC (113ºF).

3. ^{^}

   The B-59 was apparently the only submarine in the flotilla that required three officers’ approval in order to fire the “special weapon” — the others only required the captain and the political offer to approve the launch. 

^{^}

From skimming some articles and first-hand accounts, it seems unclear if the captain just had an outburst and then accurately wanted to follow protocol (and use the missile), or if he was truly reacting irrationally/emotionally because of the incredibly stressful environment. Accounts conflict a bit, and my sense is that orders around using the missile were unclear and overly permissive (or even encouraging towards using the missile).

Show all footnotes

If you feel overwhelmed by FTX-collapse-related content on the Forum, you can hide most of it by using a tag filter: hover over the "FTX collapse" tag on the Frontpage (find it to the right of the "Frontpage Posts" header), and click on "Hidden." 

[Note: this used to say "FTX crisis," and that might still show up in some places.]

Quick (visual) note on something that seems like a confusion in the current conversation:

Others have noted similar things (eg, and Will’s earlier take on total vs human extinction). You might disagree with the model (curious if so!), but I’m a bit worried that one way or another people are talking past each other (least from skimming the discussion). 

(Commenting via phone, sorry for typos or similar!)

I sometimes see people say stuff like:

Those forecasts were misguided. If they ended up with good answers, that's accidental; the trends they extrapolated from have hit limits... (Skeptics get Bayes points.)

But IMO it's not a fluke that the "that curve is going up, who knows why" POV has done well. 

A sketch of what I think happens:

There’s a general dynamic here that goes something like:

1. Some people point to a curve going up (and maybe note some underlying driver)
2. Others point out that the drivers have inherent constraints (this is an s-curve, not an exponential)

And then in *some* sense the bottlenecks crowd turn out to be right (the specific drivers/paradigm peters out, there’s literally no more space for more transistors, companies run low on easily accessible/high quality training data, etc.)…

…but then a "surprise new thing" pops up and fills the gap, such that the “true” thing we cared about (whether or not it’s what we were originally measuring) *does* actually continue as people originally predicted, apparently naively 

(and it turned out that the curve consists of a stack of s-curves..)

We can go too far with this kind of reasoning; some “true things we care about” (e.g. spread of a disease) * are* in fact s-curves, bounded, etc., and only locally look like ~exponentials. (So: no, we shouldn't expect the baby to weigh trillions of pounds by age 10...)

But I think the more granular, gears-oriented view — which considers how long specific drivers of the progress we're seeing could continue, etc. — often underrates the extent to which *other forces* can (and often do) jump in when earlier drivers lose momentum.

(~Crossposting from Twitter.)

"The Bypass Principle: How AI flows around obstacles" from Eric Drexler is a very related (and IMO good) post. Quote (bold mine):

While shallow assessments focus on visible obstacles — the difficulties of matching human capabilities, of overcoming regulatory barriers, and of restructuring organizations — AI-enabled developments will often find paths that bypass rather than overcome apparent barriers. Existing obstacles are concrete and obvious in a way that alternatives are not. Skewed judgment follows.

(This stuff isn't new; many people have pointed out these kinds of dynamics. But I feel like I'm still seeing them a fair bit — and this came up recently — so I wanted to write this note.)

I keep coming back to this map/cartogram. It's just so great. 

Superman gets to business [private submission to the Creative Writing Contest from a little while back]

“I don’t understand,” she repeated. “I mean, you’re Superman.”

“Well yes,” said Clark. “That’s exactly why I need your help! I can’t spend my time researching how to prioritize while I should be off answering someone’s call for help.”

“But why prioritize? Can’t you just take the calls as they come?”

Lois clicked “Send” on the email she’d been typing up and rejoined the conversation. “See, we realized that we’ve been too reactive. We were taking calls as they came in without appreciating the enormous potential we had here. It’s amazing that we get to help people who are being attacked, help people who need our help, but we could also make the world safer more proactively, and end up helping even more people, even better, and when we realized that, when that clicked—”

“We couldn’t just ignore it.”

Tina looked back at Clark. “Ok, so what you’re saying is that you want to save people— or help people — and you think there are better and worse ways you could approach that, but you’re not sure which are which, and you realized that instead of rushing off to fight the most immediate threat, you want to, what, do some research and find the best way you can help?”

“Yes, exactly, except, they’re not just better, we think they might be seriously better. Like, many times better. The difference between helping someone who’s being mugged, which by the way is awful, so helping them is already pretty great, but imagine if there’s a whole city somewhere that needs water or something, and there are people dying, and I could be helping them instead. It’s awful to ignore the mugging, but if I’m going there, I’m ignoring the city, and of those...”

“Basically, you’re right, Tina, yes,” said Lois.

“Ok,” Tina felt like she was missing something. “But Lois, you’re this powerful journalist, and Clark, you’re Superman. You can read at, what, 100 words per second? Doesn’t it make more sense for you to do the research? I’d need to spend hours reading about everything from food supply chains in Asia to, I dunno, environmental effects of diverting rivers or something, and you could have read all the available research on this in a week.”

“It’s true, Clark reads fast, and we were even trying to split the research up like that at some point,” said Lois. “But we also realized that the time that Clark was spending reading, even if it wasn’t very long, he could be spending chasing off the villain of the week or whatever. And I couldn’t get to all the research in time. I tried for a while, but I have a job, I need to eat, I need to unwind and watch Spanish soap operas sometimes. I was going insane. So we’ve been stuck in this trap of always addressing the most urgent thing, and we think we need help. Your help.”

“Plus, we don’t even really know what we need to find out. I don’t know which books I should be reading. It’s not even just about how to best fix the problem that’s coming up, like the best way to help that city without water. It’s also about finding new problems. We could be missing something huge.”

“You mean, you need to find the metaphorical cities without water?” Clark was nodding. Lois was tapping out another email. “And you should probably be widening your search, too. Not just looking at people specifically, or looking for cities without water, but also looking for systems to improve, ways to make people healthier. Animals, too, maybe. Aliens? Are there more of you? I’m getting off track.” Tina pulled out the tiny notebook her brother gave her and began jotting down some questions to investigate.

“So, are you in?” Lois seemed a bit impatient. Tina set the notebook aside, embarrassed for getting distracted.

“I think so. I mean, this is crazy, I need to think about it a bit. But it makes sense. And you need help. You definitely shouldn’t be working as a journalist, Clark. I mean, not that I’m an expert, really, but—”

“You kind of are. The expert.” Tina absently noted that Clark perfectly fit her mental image of a proper Kansas farm boy. He was even wearing plaid.

“If you accept the offer.” Lois said, without looking up from her email.

“That’s a terrifying thought. It feels like there should be more people helping, here. You should have someone sanity-checking things. Someone looking for flaws in my reasoning. You should maybe get a personal assistant, too— that could free up a massive amount of your time, and hopefully do a ton of good.” Tina knew she was hooked, but wanted to slow down, wanted to run this whole situation by a friend, or maybe her brother. “Can I tell someone about this? Like, is all of this secret?”

Clark shook his head. “We don’t want to isolate you from your friends or anything. But there will be things that need to be secret. And we’ve had trouble before— secrets are hard—” Clark glanced apologetically at Lois, who looked up from her frantic typing for long enough to shoot him a look, “But as much as possible, we don’t want to fall into bad patterns from the past.”

“I guess there are some dangers with information leaking. You probably have secret weaknesses, or maybe you know things that are dangerous—” Tina’s mind was swirling with new ideas and new worries. “Wait a second, how did you even find me? How do you know I’m not going to, like, tell everyone everything...”

Clark and Lois looked at each other.

“We didn’t really think that through very much. You seemed smart, and nice, and you’d started that phone-an-anonymous-friend service in college. And you wrote a good analysis when we asked you to. Sorry about the lie about the consulting job, by the way.”

“And you really need help.” Tina nodded. “Ok, we definitely need to fine-tune the hiring process. And I’ll start by writing down a list of some key questions.”

“I’ll order takeout,” said Lois, and pulled out her phone. 

[I wrote and submitted this shortly before the deadline, but was somewhat overwhelmed with other stuff and didn't post it on the Forum. I figured I'd go ahead and post it now. (Thanks to everyone who ran, participated in, or encouraged the contest by reading/commenting!]

I recently ran a quick Fermi workshop, and have been asked for notes several times since. I've realized that it's not that hard for me to post them, and it might be relatively useful for someone.

Quick summary of the workshop

1. What is a Fermi estimate?
2. Walkthrough of the main steps for Fermi estimation
   1. Notice a question
   2. Break it down into simpler sub-questions to answer first
   3. Don’t stress about the details when estimating answers to the sub-questions
   4. Consider looking up some numbers
   5. Put everything together
   6. Sanity check
3. Different models: an example
4. Examples!
5. Discussion & takeaways

Resources

• Guesstimate is a great website for Fermi estimation (although you can also use scratch paper or spreadsheets if that's what you prefer)
• This is a great post on Fermi estimation
• In general, you can look at a bunch of posts tagged "Fermi Estimation" on LessWrong or look at the Forum wiki description

Disclaimers:

1. I am not a Fermi pro, nor do I have any special qualifications that would give me credibility :)
2. This was a short workshop, aimed mostly at people who had done few or no Fermi estimates before

***************

***************

***************

***************

***************

***************

***************

***************

***************

***************