"Starting a company is like chewing glass. Eventually, you start to like the taste of your own blood."
Building a new organization is extremely hard. It's hard when you've done it before, even several times. It's even harder the first time.
Some new organizations are very similar to existing organizations. The founders of the new org can go look at all the previous closeby examples, learn from them, copy their playbook and avoid their mistakes. If your org is shaped like a Y-combinator company, you can spend dozens of hours absorbing high-quality, expert-crafted content which has been tested and tweaked and improved over hundreds of companies and more than a decade. You can do a 15 minute interview to go work next to a bunch of the best people who are also building your type of org, and learn by looking over their shoulder and troubleshooting together. You get to talk to a bunch of people who have actually succeeded building an org-like-yours.
How likely is org building success, in this premier reference class, rich with prior examples to learn from, with a tried and true playbook, a tight community of founder peers, the advice of many people who have tried to do your kind of thing and won?
5%.
https://pitchbook.com/news/articles/y-combinator-accelerator-success-rate-unicorns
An AI safety lab is not the same as a Y-combinator company.
It is. WAY. FUCKING. HARDER.
Y-combinator crowd has a special category for orgs which are trying build something that requires > ~any minor research breakthrough: HARD tech.
Yet the vast majority of these Hard Tech companies are actually building on top of an academic field which basically has the science figured out. Ginkgo Bioworks did not need to figure out the principles of molecular biology, nor the tools and protocols of genetic engineering. They took the a decades old, well-developed paradigm, and worked within it to incrementally build something new.
How does this look for AI safety?
And how about timing. Y-combinator reference class companies take a long time to build. Growing headcount slowly, running lean: absolutely essential if you are stretching out your last funding round over 7 years to iterate your way from a 24 hour livestream tv show of one guy's life to a game streaming company.
Remind me again, what are your timelines?
I could keep going on this for a while. People? Fewer. Funding? Monolithic. Advice from the winners? HA.
Apply these updates to our starting reference class success rate of
ONE. IN. TWENTY.
Now count the AI safety labs.
Multiply by ~3.
That is the roughly the number of people who are not the subject of this post.
For all the rest of us, consider several criticisms and suggestions, which were not feasible to run by the subjects of this post before publication
0. Nobody knows what they are fucking doing when founding and running an AI safety lab and everyone who says they do is lying to you.
1. Nobody has ever seen an organization which has succeeded at this goal.
2. Nobody has ever met the founder of such an organization, nor noted down their qualifications.
3. If the quote at the top of this post doesn't evoke a visceral sense memory for you, consider whether you have an accurate mental picture of what it looks like and feels like to be succeeding at this kind of thing from the inside. Make sure you imagine having fully internalized that FAILURE IS YOUR FAULT and no one else's, and are defining success correctly. (I believe it should be "everyone doesn't die" rather than "be highly respected for your organization's contributions" or "avoid horribly embarrassing mistakes".)
4. If that last bit feels awful and stress inducing, I expect that is because it is. Even for and especially for the handfulls of people who are not the subjects of this post. So much so that I'm guessing that whatever it is that allows people to say "yes" to that responsibility is the ~only real qualification to adding a one to the number of AI safety labs we counted earlier.
5. You have permission. You do not need approval. You are allowed to do stupid things, have no relevant experience, be an embarrassing mess, and even ~*~fail to respond criticism~*~
6. Some of us know what it looks like to be chewing glass, and we have tasted our own blood. We know the difference between the continuous desperate dumpster fires and the real mistakes. We will be silently cheering you through the former and grieving with you on the latter. Sometimes we will write you a snarky post under a pseudonym when we really should be sleeping.
522 companies went through Y-combinator over the last year. Imagine that.
Thank you for reading this loveletter to the demeaning occupation of desperately trying. It's addressed to you, if you'd like.
I am confused about what your claims are, exactly (or what you’re trying to say).
One interpretation, which makes sense to me, is the following
I really like and appreciate this point. Speaking for me personally, I too often fall into the trap of criticising someone for doing something not perfectly and not 1. Appreciating that they have tried at all and that it was potentially really hard, and 2. Criticising all the people who didn’t do anything and chose the safe route. There is a good post about this: Invisible impact loss (and why we can be too error-averse).
In addition, I think it could be a valid point to say that we should be more understanding if e.g. the research agendas of AIS labs are/were off in the past as this is a problem that no one really knows how to solve and that is just very hard. I don’t really feel qualified to comment on that.
Your post could also be claiming something else:
For instance, you seem to claim that the reference class of people who can advise people working on AI safety is some group whose size is the number of AI safety labs multiplied by 3. (This is what I understand your point to be if I look at the passage that starts with “Some new organizations are very similar to existing organizations. The founders of the new org can go look at all the previous closeby examples, learn from them, copy their playbook and avoid their mistakes.” and ends in “That is the roughly the number of people who are not the subject of this post.”)
If this is what you want to say, I think the message is wrong in important ways. In brief:
1. A claim like: “We should not criticise / should have a very high bar for criticizing AI safety labs / their founders (especially not if you yourself have not started an AIS lab).”
As stated above, I think it is important to appreciate people for trying at all, and it’s useful to notice that work not getting done is a loss. That being said, criticism is still useful. People are making mistakes that others can notice. Some organizations are less promising than others, and it’s useful to make those distinctions so that we know which to work in or donate to.
In a healthy EA/LT/AIS community, I want people to criticise other organisations, even if what they are doing is very hard and has never been done before. E.g. you could make the case that what OP, GiveWell, and ACE are doing has never been done before (although it is slightly unclear to me what exactly “doing something that has never been done before” means), and I don’t think anyone would say that those organisations should be beyond criticism.
This ties nicely into the second point I think is wrong:
2. A claim like: “they’re doing something that no one else has done before … they don’t have anyone to learn from”
A quote from your post:
A point I think you’re making:
“They are doing something that no one else has done before [build a successful AI safety lab], and therefore, if they make mistakes, that is way understandable because they don’t have anyone to learn from.”
It is true that the closer your organisation is to an already existing org/cluster of orgs, the more you will be able to copy. But just because you’re working on something new that no one has worked on (or your work is different in other important aspects), it doesn’t mean that you cannot learn from other organisations, their successes and failures. For things like having a healthy work culture, talent retention, and good governance structures, there are examples in the world that even AIS labs can learn from.
I don’t understand the research side of things well enough to comment on whether/how much AIS labs could learn from e.g. academic research or for-profit research labs working on problems different from AIS.
I don't expect I'll manage to rewrite this post in the way which makes everything I believe clear (and I'm not sure that would be very valuable for others if I did)