All of Nikola's Comments + Replies

No, it's closer to 2030 now.

Some successor species are much "worthier" of inheriting the future than others. I expect that the type of AI that would violently take over control of civilization would be on the less worthy side (especially conditioning on sub-2035 AGI timelines), and that the type of society we'd design given lots of deliberation and care (possibly through a long reflection) would be much more worthy.

I think I'll pass for now but I might change my mind later. As you said, I'm not sure if betting on ASI makes sense given all the uncertainty about whether we're even alive post-ASI, the value of money, property rights, and whether agreements are upheld. But thanks for offering, I think it's epistemically virtuous.

Also I think people working on AI safety should likely not go into debt for security clearance reasons.

My median is around mid 2029, largely due to business-not-as-usual scenarios like treaties, pauses, sabotage, and war.

Yup those conditions seem roughly right. I'd guess the cost to train will be somewhere between $30B and $3T. I'd also guess the government will be very willing to get involved once AI becomes a major consideration for national security (and there exist convincing demonstrations or common knowledge that this is true).

I'm guessing that open weight models won't matter that much in the grand scheme of things - largely because once models start having capabilities which the government doesn't want bad actors to have, companies will be required to make sure bad actors don't get access to models (which includes not making the weights available to download). Also, the compute needed to train frontier models and the associated costs are increasing exponentially, meaning there will be fewer and fewer actors willing to spend money to make models they don't profit from.

I get that it can be tricky to think about these things.

I don't think the outcomes are overdetermined - there are many research areas that can benefit a lot from additional effort, policy is high leverage and can absorb a lot more people, and advocacy is only starting and will grow enormously.

AGI being close possibly decreases tractability, but on the other hand increases neglectedness, as every additional person makes a larger relative increase in the total effort spent on AI safety.

The fact that it's about extinction increases, not decreases, the value o... (read more)

I think grant evaluators should take into account their intuitions on what kinds of research are most valuable rather than relying on expected value calculations.

In case of EV calculations where the future is part of the equation, I think using microdooms as a measure of impact is pretty practical and can resolve some of the problems inherent with dealing with enormous numbers, because many people have cruxes which are downstream of microdooms. Some think there'll be 10^40 people, some think there'll be 10^20. Usually, if two people disagree on how v... (read more)

Yup, I'd say that from the perspective of someone who wants a good AI safety (/EA/X-risk) student community, Harvard is the best place to be right now (I say this as an organizer, so grain of salt). Not many professional researchers in the area though which is sad :(

As for the actual college side of Harvard, here's my experience (as a sophomore planning to do alignment):

• Harvard doesn't seem to have up-to-date CS classes for ML. If you want to learn modern ML, you're on your own
  • (or with your friends! many HAIST people have self-studied ML together or taken

Check out this post. My views from then have slightly shifted (the numbers stay roughly the same), towards:
 

• If Earth-based life is the only intelligent life that will ever emerge, then humans + other earth life going extinct makes the EV of the future basically 0, aside from non-human Earth-based life optimizing the universe, which would probably be less than 10% of non-extinct-human EV, due to the fact that
  1. Humans being dead updates us towards other stuff eventually going extincts
  2. Many things have to go right for a species to evolve pro-social tendenci

Nice to see new people in the Balkans! I'd be down to chat sometime about how EA Croatia started off :)

Building on the space theme, I like Earthrise, as it has very hopeful vibes, but also points to the famous picture that highlights the fragility and preciousness of earth-based life.

Thank you for writing this. I've been repeating this point to many people and now I can point them to this post.

For context, for college-aged people in the US, the two most likely causes of death in a given year are suicide and vehicle accidents, both at around 1 in 6000. Estimates of global nuclear war in a given year are comparable to both of these. Given a AGI timeline of 50% by 2045, it's quite hard to distribute that 50% over ~20 years and assign much less than 1 in 6000 to the next 365 days. Meaning that even right now, in 2022, existential risks are... (read more)

Strongly agree, fostering a culture of openmindedness (love the example from Robi) and the expectation of updating from more experienced EAs seems good. In the updating case, I think making sure that everyone knows what "updating" means is a priority (sounds pretty weird otherwise). Maybe we should talk about introductory Bayesian probability in fellowships and retreats.

Great post, Joshua! I mostly second all of these points.

I'd add another hot take:

Both the return of fellowships and retreats mostly tracks one variable, and that is time participants spend in small (eg. one-on-one) interactions with highly engaged EAs. Retreats are good mostly because they're a very efficient way to have a lot of this interaction in a small period of time.  More in this here.

I agree, to clarify, my claim assumes infinite patience.

Skipping time to determine net positivity of experience

[inspired by a conversation with Robi Rahman]

Imagine that it’s possible to skip certain periods of time in your life. All this means is you don’t experience them, but you come out of them having the same memories as if you did experience them.

Now imagine that, after you live whatever life you would have lived, there’s another certain 5000 years of very good life that you’ll live that’s undoubtedly net positive. My claim is that, any moments in your life you’d prefer to “skip” are moments in which your life is net negative.

I wonder how many moments you'd skip?

I think that it's relevant that, for some veg*ns, it would take more energy (emotional energy/willpower) not to be veg*n. For instance, having seen some documentaries, I am repulsed by the idea of eating meat due to the sheer emotional force of participating in the atrocities I saw. Maybe this is an indicator that I should spend more time trying to align my emotions to my ethical beliefs (which would, without the strong emotional force, point towards me eating animal products to save energy), but I'm not sure if that's worth the effort.

Maybe this implies t... (read more)

Thanks, you're completely right, that sounds negative. Changed the title to "Helping newcomers be more objective with career choice", which probably gets across what we're trying to get across better.

Strong agree with the idea that we should emphasize actions people are taking and avoid hero-worship-like phrases. I was mostly using my own mental shorthand when I said "superhuman" and forgot to translate to other-people-speak.

Regarding the makeup of fellowship groups, I think probably giving people an option to attend some socials which are generally attended by highly engaged people could be good? So that, if there's a lack of engagement in their cohorts, they can make up for it by finding a way to interact with engaged people somewhere else.

Haven't th... (read more)

They're standard deviations, updated the figures, thanks! I agree strongly, this is weak evidence at best.