All of Joshc's Comments + Replies

The competition was cancelled. I think the funding for it was cut, though @Oliver Z can say more. I was not involved in this decision.

Thanks for this! I think it would be helpful to plot the median changes in extinction probabilities against the number of words in the article/video. I'm noticing a correlation as I click through the links and would be curious how strong it is (so this effect can be disentangled from the style of the source).

Yep, I have some ideas. Please DM me and give some info about yourself if you are interested in hearing them :)

Thanks for the referral. I agree that the distinction between serial time and parallel time is important and that serial time is more valuable. I'm not sure if it is astronomically more valuable though. There are two points we could have differing views on:

- the amount of expected serial time a successful (let's say $10 billion dollar) AI startup is likely to counterfactually burn. In the post I claimed that this seems unlikely to be more than a few weeks. Would you agree with this?
- the relative value of serial time to money (which is exchangeable with pa... (read more)

If your claim is that 'applying AI models for economically valuable tasks seems dangerous, i.e. the AIs themselves could be dangerous' then I agree. A scrappy applications company might be more likely to end the world than OpenAI/DeepMind... it seems like it would be good, then, if more of these companies were run by safety conscious people.

A separate claim is the one about capabilities externalities. I basically agree that AI startups will have capabilities externalities, even if I don't expect them to be very large. The question, then, is how much expected money we would be trading for expected time and what is the relative value between these two currencies.

It's unclear to me that having EA people starting an AI startup is more tractable than convincing other people that the work is worth funding

Yeah, this is unclear to me to. But you can encourage lots of people to pursue earn-to-give paths (maybe a few will succeed). Not many are in a position to persuade people, and more people having this as an explicit goal seems dangerous.

Also, as an undergraduate student with short timelines, the startup path seems like a better fit.

I don't see how the flexibility of money makes any difference? Isn't it frustratingly d

That's a good point. Here's another possibility:
Require that students go through a 'research training program' before they can participate in the research program. It would have to actually help prepare them for technical research though. Relabeling AGISF as a research training program would be misleading, so you would want to add a lot more technical content (reading papers, coding assignments, etc.)  It would probably be pretty easy to gauge how much the training program participants care about X-risk / safety and factor that in when deciding whethe... (read more)

Oo exciting. Yeah, the research program looks like it is closer to what I'm pitching. 

Though I'd also be excited about putting research projects right at the start of the pipeline (if they aren't already). It looks like AGISF is still at the top of your funnel and I'm not sure if discussion groups like these will be as good for attracting talent.

Late to the party here, but I was wondering why these organizations need aligned engineering talent. Anthropic seems like the kind of org that talented, non-aligned people would be interested in...

These are reasonable concerns, thanks for voicing them. As a result of unforeseen events, we became responsible for running this iteration only a couple of weeks ago. We thought that getting the program started quickly — and potentially running it at a smaller scale as a result — would be better than running no program at all or significantly cutting it down.

The materials (lectures, readings, homework assignments) are essentially ready to go and have already been used for MLSS last summer. Course notes are supplementary and are an ongoing project.

We are pu... (read more)

Yes! Thanks for asking.

Fixed, thanks!

Yeah, I would be in favor of interaction in simulated environments -- other's might disagree, but I don't think this influences the general argument very much as I don't think leaving some matter for computers will reduce the number of brains by more than an order of magnitude or so.

Having a superintelligence aligned to normal human values seems like a big win to me! 

Not super sure what this means but the 'normal human values' outcome as I've defined it hardly contributes to EV calculations at all compared to the utopia outcome. If you disagree with this, please look at the math and let me know if I made a mistake.

Yep, I didn't initially understand you. That's a great point!

This means the framework I presented in this post is wrong. I agree now with your statement:

the EV of partly utilitarian AI is higher than that of fully utilitarian AI.

I think the framework in this post can be modified to incorporate this and the conclusions are similar. The quantity that dominates the utility calculation is now the expected representation of utilitarianism in the AGI's values.

The two handles become:
(1) The probability of misalignment.
(2) The expected representation of utilitaria... (read more)

Yep, thanks for pointing that out! Fixed it.

...I haven't seen much discussion about the downsides of delaying

I'm not sure how your first point relates to what I was saying in this post; but, I'll take a guess. I said something about how investing in capabilities at anthropic could be good. An upside to this would be increasing the probability that EAs end up controlling the super-intelligent AGI in the future. The downside is that it could shorten timelines, but hopefully this can be mitigated by keeping all of the research under wraps (which is what they ... (read more)

These are great! I'll add that you should be careful not to overbook yourself. I would leave an hour and a half in the middle of the day open in case you want to take a nap.

This could be helpful. Maybe posting questions on the EA forum and allowing the debate to happen in the comments could be a good format for this.

Got it! I edited the point about in-person reading so that it provides a more accurate portrayal of what you all are doing.