Not everything being funded here even IS alignment techniques, but also, insofar as you just want general better understanding of AI as a domain through science, why wouldn't you learn useful stuff from applying techniques to current models. If the claim is that current models are too different from any possible AGI for this info to be useful, why do you think "do science" would help prepare for AGI at all? Assuming you do think that, which still seems unclear to me. 

I asked about genuine research creativity not AGI, but I don't think this conversation is going anywhere at this point. It seems obvious to me that "does stuff mathematicians say makes up the building blocks of real research" is meaningful evidence that the chance that models will do research level maths in the near future is not ultra-low, given that capabilities do increase with time. I don't think this analogous to IQ tests or the bar exam, and for other benchmarks, I would really need to see what your claiming is the equivalent of the transfer from frontier math 4 to real math that was intuitive but failed. 

The forum is kind of a bit dead generally, for one thing. 

I don't really get on what grounds your are saying that the Coefficient Grants are not to people to do science, apart from the governance ones. I also think you are switching back and forth between: "No one knows when AGI will arrive, best way to prepare just in case is more normal AI science" and "we know that AGI is far, so there's no point doing normal science to prepare against AGI now, although there might be other reasons to do normal science." 

I guess I still just want to ask: If models hit 80% on frontier math by like June 2027, how much does that change your opinion on whether models will be capable of "genuine creativity" in at least one domain by 2033. I'm not asking for an exact figure, just a ballpark guess. If the answer is "hardly at all", is there anything short of an 100% clear example of a novel publishable research insight in some domain, that would change your opinion on when "real creativity" will arrive? 

I think what you are saying here is mostly reasonable, even if I am not sure how much I agree: it seems to turn on very complicated issue in the philosophy of probability/decision theory, and what you should do when accurate prediction is hard, and exactly how bad predictions have to be to be valueless. Having said that, I don't think your going to succeed in steering conversation away from forecasts if you keep writing about how unlikely it is that AGI will arrive near term. Which you have done a lot, right? 

I'm genuinely not sure how much EA funding for AI-related stuff even is wasted on your view. To a first approximation, EA is what Moskowitz and Tuna fund. When I look at Coefficient's-i.e. what previously was Open Phil's-7 most recent AI safety and governance grants here's what I find: 

1) A joint project of METR and RAND to develop new ways of assessing AI systems for risky capabilities.

2) "AI safety workshop field building" by BlueDot Impact

3) An AI governance workshop at ICML 

4) "General support" for the Center for Governance of AI. 

5) A "study on encoded reasoning in LLMs at the University of Maryland"

6) "Research on misalignment" here: https://www.meridiancambridge.org/labs 

7) "Secure Enclaves for LLM Evaluation" here https://openmined.org/

So is this stuff bad or good on the worldview you've just described? I have no idea, basically. None of it is forecasting, plausibly it all broadly falls under either empirical research on current and very near future models, training new researchers, or governance stuff, though that depends on what "research on misalignment" means. But of course, you'd only endorse if it is good research. If you are worried about lack of academic credibility specifically, as far as I can tell 7 out of the 20 most recent grants are to academic research in universities. It does seem pretty obvious to me that significant ML research goes on at places other than universities, though, not least the frontier labs themselves. 
 

I guess I feel like if being able to solve mathematical problems designed by research mathematicians to be similar to the kind of problems they solve in their actual work is not decent evidence that AIs are on track to be able to do original research in mathematics in less than say 8 years then what would you EVER accept as empirical evidence that we are on track for that, but not there yet?  

Note that I am not saying this should push your overall confidence to over 50% or anything, just that it ought to move you up by a non-trivial amount relative to whatever your credence was before. I am certainly NOT saying that skill on Frontier Math 4 will inevitably transfer to real research mathematics, just that you should think there is a substantial risk that it will. 

I am not persuaded by the analogy to IQ test scores for the following reason. It is far from clear that the tasks that LLMs can't do despite scoring 100 on IQ tests are anything like as similar as the Frontier Math 4 tasks are at least allegedly designed to resemble real research questions in mathematics*, because the latter are being deliberately designed for similarity, whereas IQ tests are just designed so that skill on them correlates with skill on intellectual tasks in general among humans. (I also think the inference towards "they will be able to DO research math", from progress on Frontier Math 4, is rather less shaky than "they will DO proper research math in the same way as humans". It's not clear to me what tasks actually require "real creativity" if that means a particular reasoning style, rather than just the production of novel insights as an end product. I don't think you or anyone else knows this either.) Real math is also uniquely suited to questions-and-answer benchmarks I think, because things really are often posed as extremely well-defined problems with determinate answers, i.e. prove X. Proving things is not literally the only skill mathematicians have, but being able to prove the right stuff is enough to be making a real contribution. In my view that makes claims for construct validity here much more plausible than say, inferring Chat-GTP can be a lawyer if it passes the bar exam. 

In general, your argument here seems like it could be deployed against literally any empirical evidence that AIs were approaching being able to do a task, short of them actually performing that task. You can always say "just because in humans, ability to do X is correlated with ability to do Y, doesn't mean the techniques the models are using to do X can do Y with a bit of improvement." And yes, that is always true, that it doesn't *automatically* mean that. But if you allow this to mean that no success on any task ever significantly moves you at all about future real world progress on intuitively similar but harder tasks, you are basically saying it is impossible to get empirical evidence that progress is coming before it has arrived, which is just pretty suspicious a priori. What you should do in my view, is think carefully about the construct validity of the particular benchmark in question, and then-roughly-updated your view based on how likely you think it is to be basically valid, and what it would mean if it was. You should take into account the risk that success on Frontier Math 4 is giving real signal, not just the risk that it is meaningless. 

My personal guess is that it is somewhat meaningful, and we will see the first real AI contributions to maths in 6-7 years, that is 60% chance by then of AI proofs important enough for credible mid-ranking journals. EDIT: I forgot my own forecast here, I expect saturation in about 5 years so "several" years is an exaggeration. Nonetheless I expect some gap between Frontier Math 4 being saturated and the first real contribuitions to research mathematics: I guess 6-9 years until real contributions is more like my forecast than 6-7 To be clear, I say "somewhat" because this is several years after I expect the benchmark itself to saturate.  But I am not shocked if someone thinks "no, it is more likely to be meaningless". But I do think if your going to make a strong version of the "it's meaningless" case where you don't see the results as signal to any non-negligible degree, you need more than to just say "some other benchmarks in far less formal demains, apparently far less similar to the real world tasks being measured, have low construct validity." 

 In your view, is it possible to design a benchmark that a) does not literally amount to "produce a novel important proof", but b) nonetheless improvements on the benchmark give decent evidence that we are moving towards models being able to do this? If it is possible, how would it differ from Frontier Math 4? 

*I am prepared to change my mind on this if a bunch of mathematicians say "no, actually the questions don't look like they were optimized for this." 

 

"Rob Wiblin opines that the fertility crash would be a global priority if not for AI likely replacing human labor soon and obviating the need for countries to have large human populations"

This is a case where it really matters whether you are giving an extremely high chance that AGI is coming within 20-30 years, or merely a decently high chance. If you think the chance is like 75%, and the claim that conditional on no AGI, low fertility would be a big problem is correct, then the problem is only cut by 4x, which is compatible with it still being large and worth working on. Really, you need to get above 97-8% before it starts looking clear that low fertility is not worth worrying about, if we assume that conditional on no AGI it will be a big problem. 

I'm not actually that interested in defending:

1. The personal honor of Yudkowsky, who I've barely read and don't much like, or his influence on other people's intellectual style. I am not a rationalist, though I've met some impressive people who probably are.
2. The specific judgment calls and arguments made in AI 2027.
3. Using the METR graph to forecast superhuman coders (even if I probably do think this is MORE reasonable than you do; but I'm not super-confident about its validity as a measure of real-world coding. But I was not trying to describe how I personally would forecast superhuman coders, but just to give a hypothetical case where making a forecast more "subjective" plausibly improves it.)

Rather what I took myself to be saying was:

1. Judgmental forecasting is not particularly a LW thing, and it is what AI2027 was doing, whether or not they were doing it well.
2. You can't really avoid what you are calling "subjectivity" when doing judgmental forecasting, at least if that means not just projecting a trend in data and having done with it, but instead letting qualitative considerations effect the final number you give.
3. Sometimes it would clearly make a forecast  better to make it more "subjective" if that just means less driven only by a projection of a trend in data into the future.
4. In predicting a low chance of AGI in the near term, you are also just making an informed guess influenced by data but also by qualitative considerations, argumemt, gut instinct etc. At that level of description, your forecast is just as "made up" as AI2027. (But of course this is completely compatible with the claim that some of AI2027's specific guesses are not well-justified enough or implausible.)

Now, it may be that forecasting is useless here, because no one can predict how technology will develop five years out. But I'm pretty comfortable saying that if THAT is your view, then you really shouldn't also be super-confident the chance of near-term AGI is low. Though I do think saying "this just can't be forecasted reliably" on its own is consistent with criticizing people who are confident AGI is near.

My thought process didn't go beyond "Yarrow seems committed to a very low chance of AI having real, creative research insights in the next few years, here is something that puts some pressure on that". Obviously I agree that when AGI will arrive is a different question from when models will have real insights in research mathematics. Nonetheless I got the feeling-maybe incorrectly, that your strength of conviction that AGI is partly based on things like "models in the current paradigm can't have 'real insight'", so it seemed relevant, even though "real insight in maths is probably coming soon, but AGI likely over 20 years away" is perfectly coherent, and indeed close to my own view. 

Anyway, why can't you just answer my question? 

Working on AI isn't the same as doing EA work on AI to reduce X-risk. Most people working in AI are just trying to make the AI more capable and reliable. There probably is a case for saying that "more reliable" is actually EA X-risk work in disguise, even if unintentionally, but it's definitely not obvious this is true.