Some successor species are much "worthier" of inheriting the future than others. I expect that the type of AI that would violently take over control of civilization would be on the less worthy side (especially conditioning on sub-2035 AGI timelines), and that the type of society we'd design given lots of deliberation and care (possibly through a long reflection) would be much more worthy.

I think I'll pass for now but I might change my mind later. As you said, I'm not sure if betting on ASI makes sense given all the uncertainty about whether we're even alive post-ASI, the value of money, property rights, and whether agreements are upheld. But thanks for offering, I think it's epistemically virtuous.

Also I think people working on AI safety should likely not go into debt for security clearance reasons.

My median is around mid 2029, largely due to business-not-as-usual scenarios like treaties, pauses, sabotage, and war.

Yup those conditions seem roughly right. I'd guess the cost to train will be somewhere between $30B and $3T. I'd also guess the government will be very willing to get involved once AI becomes a major consideration for national security (and there exist convincing demonstrations or common knowledge that this is true).

I'm guessing that open weight models won't matter that much in the grand scheme of things - largely because once models start having capabilities which the government doesn't want bad actors to have, companies will be required to make sure bad actors don't get access to models (which includes not making the weights available to download). Also, the compute needed to train frontier models and the associated costs are increasing exponentially, meaning there will be fewer and fewer actors willing to spend money to make models they don't profit from.

I get that it can be tricky to think about these things.

I don't think the outcomes are overdetermined - there are many research areas that can benefit a lot from additional effort, policy is high leverage and can absorb a lot more people, and advocacy is only starting and will grow enormously.

AGI being close possibly decreases tractability, but on the other hand increases neglectedness, as every additional person makes a larger relative increase in the total effort spent on AI safety.

The fact that it's about extinction increases, not decreases, the value of marginally shifting the needle. Working on AI safety saves thousands of present human lives on expectation.

I think grant evaluators should take into account their intuitions on what kinds of research are most valuable rather than relying on expected value calculations.

In case of EV calculations where the future is part of the equation, I think using microdooms as a measure of impact is pretty practical and can resolve some of the problems inherent with dealing with enormous numbers, because many people have cruxes which are downstream of microdooms. Some think there'll be 10^40 people, some think there'll be 10^20. Usually, if two people disagree on how valuable the long-term future is, they don't have a common unit of measurement for what to do today. But if they both use microdooms, they can compare things 1:1 in terms of their effect on the future, without having to flesh out all of the post-agi cruxes.

Yup, I'd say that from the perspective of someone who wants a good AI safety (/EA/X-risk) student community, Harvard is the best place to be right now (I say this as an organizer, so grain of salt). Not many professional researchers in the area though which is sad :(

As for the actual college side of Harvard, here's my experience (as a sophomore planning to do alignment):

• Harvard doesn't seem to have up-to-date CS classes for ML. If you want to learn modern ML, you're on your own
  • (or with your friends! many HAIST people have self-studied ML together or taken MLAB)
• Grade inflation is huge. You can get most of a degree by doing around 15-20 hours of schoolwork a week if you half-ass it with everything you've got
• You can get credit for alignment upskilling through independent study, making your non-alignment workload even smaller. I'm planning to do this at some point and might have thoughts later
• There are some great linear algebra and probability classes at Harvard, both of which are very useful for AI safety
• Prereqs seem super flexible most of the time. I've applied to at least 2 or 3 classes without having the formal prereqs in place and  a few sentences describing my experience were enough to get me in every time.
• There are some required classes (such as a set of  a few GENED courses) which will probably not be very useful for alignment, but you can make all of them either fun or basically zero-effort. One of them, Evolving Morality: From Primordial Soup to Superintelligent Machines, is partly about AI safety and it's great! Strongly recommend taking it at some point.

If community building potential is part of your decision process, then I would consider not going to Harvard, as there are a bunch of people there doing great things. MIT/Stanford/other top unis in general seem much more neglected in that regard, so if you could see yourself doing communty building I'd keep that in mind.

Check out this post. My views from then have slightly shifted (the numbers stay roughly the same), towards:
 

• If Earth-based life is the only intelligent life that will ever emerge, then humans + other earth life going extinct makes the EV of the future basically 0, aside from non-human Earth-based life optimizing the universe, which would probably be less than 10% of non-extinct-human EV, due to the fact that
  1. Humans being dead updates us towards other stuff eventually going extincts
  2. Many things have to go right for a species to evolve pro-social tendencies in the way humans did, meaning it might not happen before the Earth becomes uninhabitable
•  This implies we should worry much more about X-Risks to all of Earth life (misaligned AI, nanotech) per unit of probablity than X risks to just humanity, due to the fact that all of Earth life dying would mean that the universe is permanently sterilized of value, while some other species picking up the torch would preserve some possibility of universe optimization, especially in worlds where CEV is very consistent across Earth life

• If Earth-based life is not the only intelligent life that will ever emerge, then the stakes become much lower because we'll only get our allotted bubble anyways, meaning that
  1. If humans go extinct, then some alien species will eventually grab our part of space
  2. Then the EV of the universe (that we can affect) is roughly bounded by how much big our bubble is (even including trade, becasue the most sensible portion of  a trade deal is proportional to bubble size), which is probably on the scale of tens of thousands to billions of light-years(?) wide, bounding our portion of the universe to probably less than 1% of the non-alien scenario
• This implies that we should care roughly equally about human-bounded and Earth-bounded X-risks per unit of probability, as there probably wouldn't be time for another Earth species to pick up the torch between the time humans go extinct and the time Earth makes contact with aliens (at which point it's game over)

Nice to see new people in the Balkans! I'd be down to chat sometime about how EA Croatia started off :)