All of Tom_Davidson's Comments + Replies

Is the threat model for stages one and two the same as the one in my post on whether one country can outgrow the rest of the world?

That post just looks at the basic economic forces driving super-exponential growth and argues that they would lead an actor that starts with more than half of resources, and doesn't trade, to gain a larger and larger fraction of world output. 

I can't tell from this post whether there are dynamics that are specific to settling the solar system in Stage II that feed into the first-mover advantage or whether it's ... (read more)

Perhaps many minds end up at a shared notion of what they’re aiming for, via acausal trade (getting to some grand bargain), or evidential cooperation in large worlds

This isn't convergence. It's where ppl DIDN"T converge but make a win-win compromise deal. If ppl all converge, there's no role for acausal trade/ECL 

It seems like we can predictably make moral progress by reflecting; i.e. coming to answers + arguments that would be persuasive to our former selves
I think I’m more likely to update towards the positions of smart people who’ve thought long and hard about a topic than the converse (and this is more true the smarter they are, the more they’re already aware of all the considerations I know, and the more they’ve thought about it)
If I imagine handing people the keys to the universe, I mostly want to know “will they put serious effort into working out what the r

I think my biggest uncertainty about this is:

If there were a catastrophic setback of this kind, and civilisation tried hard to save and maintain the weights of superintelligent AI (which they presumably would), how likely are they to succeed? 

My hunch is that they very likely could succeed. E.g. in the first cpl of decades they'd have continued access to superintelligent AI advice (and maybe robotics) from pre-existing hardware. They could use that to bootstrap to longer periods of time. E.g. saving the weights on hard drives rather than S... (read more)

Really like this post!

I'm wondering whether human-level AI and robotics will significantly decrease civilisation's susceptibility to catastrophic setbacks?

AI systems and robots can't be destroyed by pandemics. They don't depend on agriculture -- just mining and some form of energy production. And a very small number of systems could hold tacit expertise for ~all domains. 

Seems like this this might reduce the risk by a lot, such that the 10% numbers you're quoting are too high. E.g. you're assigning 10% to a bio-driven set-back. But i'd have thought that would have to happen before we get human-level robotics?

I also work at Forethought!

I agree with a lot of this, but wanted to flag that I would be very excited for ppl doing blue skies research to apply and want Forethought to be a place that's good for that. We want to work on high impact research and understand that sometimes mean doing things where it's unclear up front if it will bear fruit.

Thanks, good Q.

I'm saying that if there is such a new paradigm then we could have >10 years worth of AI progress at rates of 2020-5, and >10 OOMs of effective compute growth according to the old paradigm. But that, perhaps, within the new paradigm these gains are achieved while the efficiency of AI algs only increases slightly. E.g. a new paradigm where each doubling of compute increases capabilities as much as 1000X does today. Then, measured within the new paradigm, the algorithmic progress might seem like just a couple of OOMs so 'effective ... (read more)

Can you give more examples of where ppl are getting this wrong?

I support the 80k pivot, and the blue dot page seems ok (but yes, I'd maybe prefer smg more opinionated).

While these concerns make sense in theory I'm not sure whether it's a problem in practice

Nice!

I think that condition is equivalent to saying that A_cog explodes iff either

• phi_cog + lambda > 1 and phi_exp + lambda > 1, or
• phi_cog > 1

Where the second possibility is the unrealistic one where it could explode with just human input

Agree that i wouldn't particularly expect the efficiency curves to be the same. 

But if the phi>0 for both types of efficiency, then I think this argument will still go through.

To put it in math, there would be two types of AI software technology, one for experimental efficiency and one for cognitive labour efficiency: A_exp and A_cog. The equations are then:

dA_exp = A_exp^phi_exp F(A_exp K_res, A_cog K_inf)

dA_cog = A_cog^phi_cog F(A_exp K_res, A_cog K_inf)

And then I think you'll find that, even with sigma < 1, it explodes when phi_exp>0 and phi_cog>0.

Although note that this argument works only with the CES in compute formulation. For the CES in frontier experiments, you would have the $\frac{A{K}_{res}}{A{K}_{train}}$ so the A cancels out.

Yep, as you say in your footnote, you can choose to freeze the frontier, so you train models of a fixed capability using less and less compute (at least for a while). 

However, if $\sigma <1$, then a software-only intelligence explosion occurs only if $\varphi >1$. But if this condition held, we could get an intelligence explosion with constant, human-only research input. While not impossible, we find this condition fairly implausible. 

Hmm, I think a software-only intelligence explosion is plausible even if  $\sigma <1$, but without the implication that you can do it with human-only research input.

The basic idea is that when you double the efficiency of software, you can now:

• Run twice as many experiments
• Have

Thanks for this!

Let me try and summarise what I think is the high-level dynamic driving the result, and you can correct me if I'm confused.

CES in compute.

Compute has become cheaper while wages have stayed ~constant. The economic model then implies that:

• If compute and labour were complements, then labs would spend a greater fraction of their research budgets on labour. (This prevents labour from becoming a bottleneck as compute becomes cheaper.)

Labs aren't doing this, suggesting that compute and labour are substitutes. 

CES in frontier... (read more)

The $\sigma >1$ condition is exactly what Epoch and Forethought consider when they analyze whether the returns to research are high enough for a singularity.^[5]

Though we initially consider this, we then adjust for compute as an input to R&D and so end up considering the sigma=1 condition. It's under that condition that I think it's more likely than not that the condition for a software-only intelligence explosion holds

I like the vividness of the comparisons!

A few points against this being nearly as crazy as the comparisons suggest:

• GPT-2030 may learn much less sample efficiently, and much less compute efficiently, than humans. In fact, this is pretty likely. Ball-parking, humans do 1e24 FLOP before they're 30, which is ~20X less than GPT-4. And we learn languages/maths from way fewer data points. So the actual rate at which GPT-2030 itself gets smarter will be lower than the rates implied. 
  • This is a sense of "learn" as in "improves its own understanding". There's an

Thanks, this is a great comment.

The first and second examples seems pretty good, and useful reference points.

The third example don't seem like they are nearly as useful though. What's particularly unusual about this case is that there are two useful inputs to AI R&D -- cognitive labour and compute for experiments -- and the former will rise very rapidly but the other will not. In particular, I imagine CS departments also saw compute inputs growing in that time. And I imagine some of the developments discussed (eg proofs about algorithms) only have cogn... (read more)

•  I think utilitarianism is often a natural generalization of "I care about the experience of XYZ, it seems arbitrary/dumb/bad to draw the boundary narrowly, so I should extend this further" (This is how I get to utilitarianism.) I think the AI optimization looks considerably worse than this by default.

Why is this different between AIs and humans? Do you expect AIs to care less about experience than humans, maybe bc humans get reward during life-time learning about AIs don't get reward during in context learning?

• I can directly observe AIs and make predictions of future training methods and their values seem to result from a much more heavily optimized and precise thing with less "slack" in some sense. (Perhaps this is related to genetic bottleneck, I'm unsure.)

Can you say more about how slack (or genetic bottleneck) would affect whether AIs have values that are good by human lights?

• AIs will be primarily trained in things which look extremely different from "cooperatively achieving high genetic fitness".

They might well be trained to cooperate with other copies on tasks, if this is they way they'll be deployed in practice?

• Current AIs seem to use the vast, vast majority of their reasoning power for purposes which aren't directly related to their final applications. I predict this will also apply for internal high level reasoning of AIs. This doesn't seem true for humans.

In what sense do AIs use their reasoning power in this way? How that that affect whether they will have values that humans like?

I agree that bottlenecks like the ones you mention will slow things down. I think that's compatible with this being a "jump in forward a century" thing though.

Let's consider the case of a cure for cancer. First of all, even if it takes "years to get it out due to the need for human trials and to actually build and distribute the thing" AGI could still bring the cure forward from 2200 to 2040 (assuming we get AGI in 2035).

Second, the excess top-quality labour from AGI could help us route-around the bottlenecks you mentioned:

• Human trials: AGI might develop u

It seems to me like you disagree with Carl because you write:

• The reason for an investor to make a bet, is that they believe they will profit later
• However, if they believe in near-term TAI, savvy investors won't value future profits (since they'll be dead or super rich anyways)
• Therefore, there is no way for them to win by betting on near-term TAI

So you're saying that investors can't win from betting on near-term TAI. But Carl thinks they can win.

Local cheap production makes for small supply chains that can regrow from disruption as industry becomes more like information goods.

Could you say more about what you mean by this?

Thanks for these great questions Ben!

To take them point by point:

1. The CES task-based model incorporates Baumol effects, in that after AI automates a task the output on that task increases significantly and so its importance to production decreases. The tasks with low output become the bottlenecks to progress. 
   1. I'm not sure what exactly you mean by technological deflation. But if AI automates therapy and increases the amount of therapists by 100X then my model won't imply that the real $ value of therapy industry increases 100X. The price of therapy fall

if they had explained why their views were not moved by the expert reviews OpenPhil has already solicited.

I included responses to each review, explaining  my reactions to it. What kind of additional explanation were you hoping for?

Davidson 2021 on semi-informative priors received three reviews.

By my judgment, all three made strong negative assessments, in the sense (among others) that if one agreed with the review, one would not use the report's reasoning to inform decision-making in the manner advocated by Karnofsky (and by Beckstead).

For Hajek... (read more)

Thanks for this!

I won't address all of your points right now, but I will say that I hadn't considered that "R&D is compensating for natural resources becoming harder to extract over time", which would increase the returns somewhat. However, my sense is that raw resource extraction is a small % of GDP, so I don't think this effect would be large.

Sorry for the slow reply!

I agree you can probably beat this average by aiming specifically at R&D for boosting economic growth.

I'd be surprised if you could spend $100s millions per year and consistently beat the average by a large amount (>5X) though:

• The $2 trillion number also excludes plenty of TFP-increasing research work done by firms that don't report R&D like Walmart and many services firms.
• The broad areas where this feels most plausible to me (R&D in computing or fundamental bio-tech) are also the areas that have the biggest poten

Great question!

I would read Appendix G as conditional on "~no civilizational collapse (from any cause)", but not conditional on "~no AI-triggered fundamental reshaping of society that unexpectedly prevents growth". I think the latter would be incorporated in "an unanticipated bottleneck prevents explosive growth".

I think the question of GDP measurement is a big deal here. GDP deflators determine what counts as "economic growth" compared to nominal price changes, but deflators don't really know what to do with new products that didn't exist. What was the "price" of an iPhone in 2000? Infinity? Could this help recover Roodman's model? If ideas being produced end up as new products that never existed before, could that mean that GDP deflators should be "pricing" these replacements as massively cheaper, thus increasing the resulting "real" growth rate?

This is an int... (read more)

Thank you for this comment! I'll make reply to different points in different comments.

But then the next point seems very clear: there's been tons of population growth since 1880 and yet growth rates are not 4x 1880 growth rates despite having 4x the population. The more people -> more ideas thing may or may not be true, but it hasn't translated to more growth.

So if AI is exciting because AIs could start expanding the number of "people" or agents coming up with ideas, why aren't we seeing huge growth spurts now?

The most plausible models have dimin... (read more)

Great suggestion - thanks! Have edited.

Hey - interesting question! 

This isn't something I looked into in depth, but I think that if AI drives explosive economic growth then you'd probably see large rises in both absolute energy use and in energy efficiency.

Energy use might grow via (e.g.) massively expanding solar power to the world's deserts (see this blog from Carl Shulman). Energy efficiency might grow via replacing human  workers with AIs (allowing services to be delivered with less energy input), rapid tech progress further increasing the energy efficiency of existing goods and s... (read more)

Thanks for these thoughts! You raise many interesting points.

 On footnote 16, you "For example, the application of Laplace’s law described below implies that there was a 50% chance of AGI being developed in the first year of effort". But historically, participants in the Dartmouth conference were gloriously optimistic

I'm not sure whether the participants at Dartmouth would have assigned 50% to creating AGI within a year and >90% within a decade, as implied by the Laplace prior. But either way I do think these probabilities would have been too ... (read more)

Agreed - the framework can be applied to things other than AGI.

Thanks for this Halstead - thoughtful article.

I have a one push-back, and one question about your preferred process for applying the ITN framework.

1. After explaining the 80K formalisation of ITN you say

Thus, once we have information on importance, tractability and neglectedness (thus defined), then we can produce an estimate of marginal cost-effectiveness.

The problem with this is: if we can do this, then why would we calculate these three terms separately in the first place?

I think the answer is that in some contexts it's easier to calculate each t... (read more)

Great podcasts!

I found Nakul's article v interesting too but am surprised at what it led you to conclude.

I didn't think the article was challenging the claim that doing paradigmatic EA activities was moral. I thought Nakul was suggesting that doing them wasn't obligatory, and that the consequentialist reasons for doing them could be overridden by an individual's projects, duties and passions. He was pushing against the idea that EA can demand that everyone support them.

It seems like your personal projects would lead to do EA activities. So I'm surprised you judge EA acti... (read more)

Yeah good point.

If people choose a job which they enjoy less then that's a huge sacrifice, and should be applauded.

But EA is about doing the most good that you can.

So anyone who is doing the most good that they could possibly do is being an amazing EA. Someone on £1million who donates £50K is not doing anywhere near as much good as they could do.

The rich especially should be encouraged to make big sacrifices, as they do have the power to do the most good.

I agree completely that talking with people about values is the right way to go. Also, I don't think we need to try and convince them to be utilitarians or nearly-utilitarian. Stressing that all people are equal and pointing to the terrible injustice of the current situation is already powerful, and those ideas aren't distinctively utilitarian.

There is no a priori reason to think that the efficacy of charitable giving should have any relation whatsoever to utilitarianism. Yet it occupies a huge part of the movement.

I think the argument is that, a priori, utilitarians think we should give effectively. Further, given the facts as they far (namely that effective donations can do an astronomical amount of good), there are incredibly strong moral reasons for utilitarians to promote effective giving and thus to participate in the EA movement.

I think that [the obsession with utilitarianism] is reg

Those seem really high flow through effects to me! £2000 saves one life, but you could easily see it doing as much good as saving 600!

How are you arriving at the figure? The argument that "if you value all times equally, the flow through effects are 99.99...% of the impact" would actually seem to show that they dominated the immediate effects much more than this. (I'm hoping there's a reason why this observation is very misleading.) So what informal argument are you using?

This is a nice idea but I worry it won't work.

Even with healthy moral uncertainty, I think we should attach very little weight to moral theories that give future people's utility negligible moral weight. For the kinds of reasons that suggest we can attach them less weight don't go any way to suggesting that we can ignore them. To do this they'd have to show that future people's moral weight was (more than!) inversely proportional to their temporal distance from us. But the reasons they give tend to show that we have special obligations to people in our gen... (read more)

Great post!

Out of interest, can you give an example of an "instrumentally rational technique that require irrationality"?

Why? What are the very long term effects of a murder?

Would you similarly doubt that, on expectation, someone murdering someone else had bad consequences overall? Someone slapping you very hard in the face?

This kind of reasoning seems to bring about a universal scepticism about whether we're doing Good. Even if you think you can pin down the long term effects, you have no idea about the very long term effects (and everything else is negligible compared to very long term effects).

In defence of WALYs, and in reply to your specific points:

1. I don't share your intuition here. Well-being is what we're talking about when we say "I'm not sure he's doing so well at the moment", or when we say "I want to help people as much as possible". It's a general term for how well someone is doing, overall. It's an advantage, in my eyes, that it's not committed to any specific account of well-being, for any such account might have its drawbacks.

2. I worry that, in adopting HALYs, EA would tie its aims to a narrow view of what huma

A small quibble

One conclusion EAs might make is that their personal diets are no big deal, easily swamped as it is by the consequences of donations.

I think it's flat out wrong to conclude our diets "are no big deal". Being vegetarian for a lifetime prevents over 1000 years of animal suffering. That's a huge, huge impact.

My more serious worry is that people will draw this conclusion and eat less ethically as a result, without donating more (they already knew donating was great). But this is just psychological speculation backed up by some anecdotal evidence.

Most people who go vegetarian find its very very little effort to be 90% vegetarian after a year or so. To me this warns against the view that people will give extra because "they haven't made the sacrifice of becoming veggie". Very soon the sacrifice becomes a habit and the claim that charitable donations are affected becomes even less plausible.

I'd be interested to know if anyone has given more money because of this thread. I know that i'm more willing to eat diary products and have read others saying it made them happier eating meat.