All of David Mathers🔸's Comments + Replies

The report has many authors, some of whom maybe much less concerned or think the whole thing is silly. I never claimed that Bengio and Hinton's views were a consensus, and in any case, I was citing their views as evidence for taking the idea that AGI may arrive soon seriously, not their views on how risky AI is. I'm pretty sure I've seen them give relatively short time-lines when speaking individually, but I guess I could be misremembering. For what it's worth Yann LeCunn seems to think 10 years is about right, and Gary Marcus seems to think a guess of 10-20 years is reasonable: https://helentoner.substack.com/p/long-timelines-to-advanced-ai-have

I guess I'm just slightly confused about what economists actually think here since I'd always thought they took the idea that markets and investors were mostly quite efficient most of the time fairly seriously. 

I don't know if/how much EA money should go to AI safety either. EAs are trying to find the single best thing, and it's very hard to know what that is, and many worthwhile things will fail that bar. Maybe David Thorstad is right, and small X-risk reductions have relatively low value because another X-risk will get us in the next few centuries anyway*. What I do think is that society as a whole spending some resources caring about the risk of AGI arriving in the next ten years is likely optimal, and that it's not more silly to do so than to do many other ob... (read more)

METR has an official internal view on what time horizons correspond to "takeover not ruled out"? 

Yeah, I am inclined to agree-for what my opinion is worth which on this topic is probably not that much-that there will be many things AIs can't do even once they have a METR 80% time-horizon of say 2 days. But I am less sure of that than I am of the meta-level point about this being an important crux. 

Sure, but I I wasn't really thinking of people on LessWrong, but rather of the fact that at least some relevant experts outside of the LW milieu seem worried and/or think that AGI is not THAT far. I.e. Hinton, Bengio, Stuart Russell (for danger) and even people often quoted as skeptical experts* like Gary Marcus or Yann LeCunn often give back of the envelope timelines of 20 years, which is not actually THAT long. Furthermore I actually do think the predictions of relatively near term AGI by Anthropic and the fact that DeepMind and OpenAI have building AGI ... (read more)

I don't think this is sufficient to explain EA disinterest, because there are also neartermist EAs who are skeptical about near-term AGI, or just don't incorporate it into their assessment of cause areas and interventions. 

Somewhat surprised to hear that people can successfully pull that off. 

It seems to me like the really important thing is interpreting what "METR 80% time horizon goes to a year", or whatever endpoint you have in mind actually means. It's important if that takes longer than AI2027 predicts, obviously, but it seems more crux-y to me whether getting to that point means transformative AI is near or not, since the difference between "3 years and 7 seven years" say, while important seems less important to me than between "definitely in 7 years" and "who knows, could still be 20+ years away". 

I think part of the issue here probably is that EAs mostly don't think biodiversity is good in itself, and instead believe only humans and animals experiencing well-being is good, and that the impact on well-being of promoting biodiversity is complex, uncertain and probably varies a lot with how and where biodiversity is being promoted. Hard to try and direct biodiversity funding if you don't really clearly agree with raising biodiversity as a goal. 

Oh, ok, I agree, if the number of deer is the same after as counterfactually, it seems plausibly net positive yes.

Also, it's certainly not common sense that it is always better to have less beings with higher welfare. It's not common sense that a world with 10 incredibly happy people is better than one with a billion very slightly less happy people.

And not every theory that avoids the repugnant conclusion delivers this result, either. 

I agree, it is unclear whether welfare is actually positive. 

Those are fair point in themselves, but I don't think "less deer is fine, so long as they have a higher standard of living" has anything like the same commonsense standing as "we should protect people from malaria with insecticide even if the insecticide hurts insects". 

And it's not clear to me that assuming less deer is fine in itself even if their lives are good is avoiding taking a stance on the intractable philosophical debate, rather than just implicitly taking one side of it. 

"A potentially lower-risk example might be the warble fly (Hypoderma), which burrows under the skin of cattle and deer, causing great discomfort, yet rarely kills its host. The warble fly is small in biomass, host-specific (so doesn't greatly affect other species), and has more limited interactions beyond its host-parasite relationship. Although it does reduce the grazing and reproductive activity of hosts, these effects are comparatively minor and could be offset with non-invasive fertility control"

Remember that it's not uncontroversial that it is prefera... (read more)

And it's not so much that I think I have zero evidence: I keep up with progress in AI to some degrees, I have some idea of what the remaining gaps are to general intelligence, I've seen the speed at which capabilities have improved in recent years etc. It's that how to evaluate that evidence is not obvious, and so simply presenting a skeptic with it probably won't move them, especially as the skeptic-in this case you-probably already has most of the evidence I have anyway. If it was just some random person who had never heard of AI asking why I thought the... (read more)

Yeah, I agree that in some sense saying "we should instantly reject a theory that recommends WD" doesn't not combine super-well with belief in classical U, for the reasons you give. That's compatible with classical U's problems with WD being less bad than NU's problem's with it, is all I'm saying. 

"I'm generally against this sort of appeal to authority. While I'm open to hear the arguments of smart people, we should evaluate those arguments themselves and not the people giving them. So far, I've heard no argument that would change my opinion on this matter."

I think this attitude is just a mistake if your goal is to form the most accurate credences you can. Obviously, it is always good practice to ask people for their arguments rather than only taking what they say on trust. But your evaluation of other people's arguments is fallible, and you know it... (read more)

"It all comes down to the question of whether the current tech is relevant for ASI or not. In my estimation, it is not – something else entirely is required. The probability for us discovering that something else just now is low." 

I think Richard's idea is that you shouldn't have *super-high* confidence in your estimation here, but should put some non-negligible credence on the idea that it is wrong, and current progress is relevant. Why be close to certainty about a question that you probably think is hard and that other smart people disagree about b... (read more)

It seems like if you find it incredible to deny and he doesn't, it's very hard to make further progress :(  I'm on your side about the chance being over 1% in the next decade, I think, but I don't know how I'd prove it to a skeptic, except to gesture and say that capabilities have improved loads in a short time, and it doesn't seem like the are >20 similar sized jumps before AGI. But when I ask myself what evidence I have for "there are not >20 similar sized jumps before AGI" I come up short. I don't necessarily think the burden of proof here is... (read more)

Saying for "other thoughts on why NU doesn't recommend extinction" is a bit of a misnomer here. The Knutsson argument you've just state doesn't even try to show NU doesn't recommend extinction, it just makes a case that it is part of a wider class of more popular theories that also sometimes do this. 

An obvious response to Knutsson is that it also matters in what circumstances a theory recommends extinction, and that NU probably recommends extinction in a wider variety of circumstances where other forms of consequentialism don't, including ones where ... (read more)

Also, I notice no references to anything about the concentration of power or wealth here. Isn't that probably something we want to avoid if we want to reach a good destination, at least all things being equal?  

Even if we are bad at answering the "what would utopia look like" question, what's the reason to think we'd be any better answering the "what would viatopia look like" question? If we are just as bad or worse at answering the second question, it's either useless or actively counterproductive to switch from utopian to viatopian planning. 

N=1, but I looked at an ARC puzzle https://arcprize.org/play?task=e3721c99, and I couldn't just do it in a few minutes, and I have a PhD from the University of Oxford. I don't doubt that most of the puzzles are trivial for some humans, and some of the puzzles are trivial for most humans or that I could probably outscore any AI across the whole ARC-2 data set. But at the same time, I am a general intelligence, so being able to solve all ARC puzzles doesn't seem like a necessary criteria. Maybe this is the opposite of how doing well on benchmarks doesn't always generalize to real world tasks, and I am just dumb at these but smart overall, and the same could be true for an LLM.

 

 Gavi do vaccines, something that governments and other big bureaucratic orgs sure seem to handle well in other cases. Government funding for vaccines is how we eliminated smallpox, for example. I think "other vaccination programs" are a much better reference class for Gavi than the nebulous category of "social programs" in general. Indeed the Rossi piece you've linked to actually says "In the social program field, nothing has yet been invented which is as effective in its way as the smallpox vaccine was for the field of public health." I'm not sure i... (read more)

It can also be indeterminate over a short time who the winner of an election is because the deciding vote is being cast and plausibly there is at least some very short duration of time where it is indeterminate whether the process of that vote being cast is finished yet. It can be indeterminate how many animals were killed for food if one animal was killed for multiple reasons of which "to eat" was one reason but not the major one. Etc. etc. 

Also, pretty much every empirical outcome is potentially fuzzy in some possible situation. (Elections can be tied, whether an experience is painful and hence harm or neutral can be unclear etc.)

Thanks, I get what you meant now.

The relatively more orthodox view amongst philosophers about the heap case is roughly there is a kind of ambiguous region of successive ns where it is neither true nor false that n grains make a heap. This is a very, very technical literature though, so possibly that characterization isn't quite right. None of the solutions are exactly great though, and some experts do think there is an exact "n" where some grains become a heap. 

I don't understand the bit about the IVT at all, absent further spelling out of the reasoning. 

What's the connection to the paradox? The sorites is far from trivial or solved. 

To be fair to Richard, there is a difference between a) stating your own personal probability in time of perils and b) making clear that for long-termist arguments to fail solely because they rely on time of perils, you need it to have  extremely low probability, not just low, at least if you accept the expected value theory and subjective probability estimates can legitimately be applied at all here, as you seemed to be doing for the sake of making an internal critique. I took it to be the latter that Richard was complaining your paper doesn't do.&nb... (read more)

Fair point, when I re-checked the paper, it doesn't clearly and explicitly display knowledge of the point you are making. I still highly doubt that Thorstad really misunderstands it though. I think he was probably just not being super-careful. 

I am far from sure that Thorstad is wrong that time of perils should be assigned ultra-low probability. (I do suspect he is wrong, but this stuff is extremely hard to assess.) But in my view there are multiple pretty obvious reasons why "time of Carols" is a poor analogy to "time of perils":

1. "Time of carols" is just way more specific, in a bad way than time of perils. I know that there are indefinitely many ways time of carols could happen if you get really fine-grained, but it nonetheless, intuitively, there is in some sense way more significantly differen

Obviously David, as a highly trained moral philosopher with years of engagement with EA understands how expected value works though. I think the dispute must really be about whether to assign time of perils very low credence. (A dispute where I would probably side with you if "very low" is below say 1 in 10,000). 

I think my basic reaction here is that longtermism is importantly correct about the central goal of EA if there are longtermist interventions that are actionable, promising and genuinely longtermist in the weak sense of "better than any other causes because of long-term effects", even if there are zero examples of LT interventions that meet the "novelty" criteria, or lack some significant near-term benefits. 

Firstly, I'd distinguish here between longtermism as a research program, and longtermism as a position about what causes should be prioritized ri... (read more)

Year of AGI

25 years seems about right to me, but with huge uncertainty. 

I think on the racism fron Yarrow is referring to the perception that the reason Moskowtiz won't fund rationalist stuff is because either he thinks that a lot of rationalist believe Black people have lower average IQs than whites for genetic reasons, or he thinks that other people believe that and doesn't want the hassle. I think that belief genuinely is quite common among rationalists, no? Although, there are clearly rationalists who don't believe it, and most rationalists are not right-wing extremists as  far as I can tell. 

What have EA funders done that's upset you? 

Not everything being funded here even IS alignment techniques, but also, insofar as you just want general better understanding of AI as a domain through science, why wouldn't you learn useful stuff from applying techniques to current models. If the claim is that current models are too different from any possible AGI for this info to be useful, why do you think "do science" would help prepare for AGI at all? Assuming you do think that, which still seems unclear to me. 

I asked about genuine research creativity not AGI, but I don't think this conversation is going anywhere at this point. It seems obvious to me that "does stuff mathematicians say makes up the building blocks of real research" is meaningful evidence that the chance that models will do research level maths in the near future is not ultra-low, given that capabilities do increase with time. I don't think this analogous to IQ tests or the bar exam, and for other benchmarks, I would really need to see what your claiming is the equivalent of the transfer from frontier math 4 to real math that was intuitive but failed. 

The forum is kind of a bit dead generally, for one thing. 

I don't really get on what grounds your are saying that the Coefficient Grants are not to people to do science, apart from the governance ones. I also think you are switching back and forth between: "No one knows when AGI will arrive, best way to prepare just in case is more normal AI science" and "we know that AGI is far, so there's no point doing normal science to prepare against AGI now, although there might be other reasons to do normal science." 

I guess I still just want to ask: If models hit 80% on frontier math by like June 2027, how much does that change your opinion on whether models will be capable of "genuine creativity" in at least one domain by 2033. I'm not asking for an exact figure, just a ballpark guess. If the answer is "hardly at all", is there anything short of an 100% clear example of a novel publishable research insight in some domain, that would change your opinion on when "real creativity" will arrive? 

I think what you are saying here is mostly reasonable, even if I am not sure how much I agree: it seems to turn on very complicated issue in the philosophy of probability/decision theory, and what you should do when accurate prediction is hard, and exactly how bad predictions have to be to be valueless. Having said that, I don't think your going to succeed in steering conversation away from forecasts if you keep writing about how unlikely it is that AGI will arrive near term. Which you have done a lot, right? 

I'm genuinely not sure how much EA funding... (read more)

I guess I feel like if being able to solve mathematical problems designed by research mathematicians to be similar to the kind of problems they solve in their actual work is not decent evidence that AIs are on track to be able to do original research in mathematics in less than say 8 years then what would you EVER accept as empirical evidence that we are on track for that, but not there yet?  

Note that I am not saying this should push your overall confidence to over 50% or anything, just that it ought to move you up by a non-trivial amount relative to... (read more)

"Rob Wiblin opines that the fertility crash would be a global priority if not for AI likely replacing human labor soon and obviating the need for countries to have large human populations"

This is a case where it really matters whether you are giving an extremely high chance that AGI is coming within 20-30 years, or merely a decently high chance. If you think the chance is like 75%, and the claim that conditional on no AGI, low fertility would be a big problem is correct, then the problem is only cut by 4x, which is compatible with it still being large and ... (read more)

I'm not actually that interested in defending:

1. The personal honor of Yudkowsky, who I've barely read and don't much like, or his influence on other people's intellectual style. I am not a rationalist, though I've met some impressive people who probably are.
2. The specific judgment calls and arguments made in AI 2027.
3. Using the METR graph to forecast superhuman coders (even if I probably do think this is MORE reasonable than you do; but I'm not super-confident about its validity as a measure of real-world coding. But I was not trying to describe how I personally

My thought process didn't go beyond "Yarrow seems committed to a very low chance of AI having real, creative research insights in the next few years, here is something that puts some pressure on that". Obviously I agree that when AGI will arrive is a different question from when models will have real insights in research mathematics. Nonetheless I got the feeling-maybe incorrectly, that your strength of conviction that AGI is partly based on things like "models in the current paradigm can't have 'real insight'", so it seemed relevant, even though "real ins... (read more)

Working on AI isn't the same as doing EA work on AI to reduce X-risk. Most people working in AI are just trying to make the AI more capable and reliable. There probably is a case for saying that "more reliable" is actually EA X-risk work in disguise, even if unintentionally, but it's definitely not obvious this is true. 

"Any sort of significant credible evidence of a major increase in AI capabilities, such as LLMs being able to autonomously and independently come up with new correct ideas in science, technology, engineering, medicine, philosophy, economics, psychology, etc"

Just in the spirit of pinning people to concrete claims: would you count progress on Frontier Math 4, like say, models hitting 40%*, as being evidence that this is not so far off for mathematics specifically? (To be clear, I think it is very easy to imagine models that are doing genuinely significant re... (read more)

Yeah, it's fair objection that even answer the why question like I did presupposes that EAs are wrong, or at least, merely luckily right. (I think this is a matter of degree, and that EAs overrated the imminence of AGI and the risk of takeover on average, but it's still at least reasonable to believe AI safety and governance work can have very high expected value for roughly the reasons EAs do.) But I was responding to Yarrow who does think that EAs are just totally wrong, so I guess really I was saying that "conditional on a sociological explanation being appropriate, I don't think it's as LW-driven as Yarrow thinks", although LW is undoubtedly important.)