All of hrosspet's Comments + Replies

FLI launches Worldbuilding Contest with $100,000 in prizes

That something is very unlikely doesn't mean it's unimaginable. The goal of imagining and exploring such unlikely scenarios is that with a positive vision we can at least attempt to make it more likely. Without a positive vision there are only catastrophic scenarios left. That's I think the main motivation for FLI to organize this contest.

I agree, though, that the base assumptions stated in the contest make it hard to come up with a realistic image.

A positive vision which is false is a lie. No vision meeting the contest constraints is achievable, or even desirable as a post-AGI target. There might be some good fiction that comes out of this, but it will be as unrealistic as Vinge's Zones of Thought setting. Using it in messaging would be at best dishonest, and, worse, probably self-deceptive.
Consider trying the ELK contest (I am)

During the 2 hours of reading and skimming through the relevant blog posts I was able to come up with 2 strategies (and no counter-examples so far). They seem to me as quite intuitive and easy to come up with, so I'm wondering what I got wrong about the ELK problem or the contest...

Due to the low confidence in my understanding I don't feel comfortable submitting these strategies, as I don't want to waste the ARC's team time.

My background: ML engineer (~7 years of experience), some previous exposure to AGI and AIS research and computer security.

ARC would be excited for you to send a short email to with a few bullet points describing your high level ideas, if you want to get a sense for whether you're on the right track / whether fleshing them out would be likely to win a prize.

I am pretty confident that ARC would want you to submit those strategies, especially given your background. Even if both trivially fail, it seems useful for them to  know that they did not seem obviously ruled out to you by the provided material.

Comments for shorter Cold Takes pieces

Thank you for a thought provoking post! I enjoyed it a lot.

I also find the "innovation as mining" hypothesis intuitive. I'd just add that innovation gets harder for humans, but we don't know whether it holds in general (think AI). Our mental capacity has been roughly constant since ancient Greece, but there is more and more previous work to understand before one can come up with something new.  This might not be true for AI, if their capacity scales.

On the other hand there is a combinatorial explosion of facts that you can combine to come up with an i... (read more)

Why don't governments seem to mind that companies are explicitly trying to make AGIs?

I think governments are not aware of the stop button problem and they think in case of emergency they can just shut down the company / servers running the AGI using force. That's what happened in the past with digital currencies (which Jackson Wagner mentions here as a plausible member of the same reference class as AGI for governments) before bitcoin - they either failed on their own, or if successful, were shut down by government ( 

Who are your role models?

Daniel Schmachtenberger. Look up some of his youtube interviews. I like especially the one with Lex Fridman ( He's a very thoughtful, yet humble person. His approach is very multi-disciplinary, systems-level, holistic. For me he is a role model for how he combines the world-knowledge and self-knowledge, and how clearly he is able to articulate his ideas, which I think are very EA-compatible (he mentions EA from time to time, but I haven't heard any endorsement from him). Yet he goes further than what is discussed within EA eg.... (read more)

What we learned from a year incubating longtermist entrepreneurship

Very interesting read, thanks for publishing this!

I am curious what qualified as "having longtermist experience" for you?

Glad to hear! Roughly this would mean having worked in a relevant area (e.g. bio, AI safety) for at least 1 - 2 years and able to contribute in some capacity to that field. To be clear, some ideas would require a lot more experience - this is just a rough proxy.
Matt Levine on the Archegos failure

a meaningful retrospective is much easier to come by than for, say, the Covid pandemic.

Agreed, but we have this rare example of Dominic Cummings, the chief adviser to Boris Johnson during the pandemic, being thoroughly interviewed about the UK's response to the pandemic. For me it was extremely interesting to peek under the hood of UK government departments and see their failure modes. If you enjoyed the CS report, you might enjoy this one, too.