All of the difficulty here is having the sign of your impact be positive. It's very hard to end up neutral; eg, if your work is just nonsense, it's negative because it's a distraction and attention sink. And it's quite easy to end up negative, for example, if you exaggerate the impact of your work and provide more feed into the hopium ecosystem that desperately touts up any sign of progress.
When you mess with AI, whatever you do will outweigh any other impacts of your life, of course, obviously. It's having the sign end up positive that is the hard part.
I got a one-time gift of appreciated crypto, not through MIRI, part of whose purpose as I understood it was to give me enough of a savings backstop (having in previous years been not paid very much at all) that I would feel freer to speak my mind or change my mind should the need arise.
I have of course already changed MIRI's public mission sharply on two occasions, the first being when I realized in 2001 that alignment might need to be a thing, and said so to the primary financial supporter who'd previously supported MIRI (then SIAI) on the premise of charging straight ahead on AI capabilities; the second being in the early 2020s when I declared publicly that I did not think alignment technical work was going to complete in time and MIRI was mostly shifting over to warning the world of that rather than continuing to run workshops. Should I need to pivot a third time, history suggests that I would not be out of a job.
Okay; I guess I was confused by your question because I thought I'd said that in the main doc.
To repeat and with added explanation: Only opinions from before ChatGPT count.
This is because ChatGPT moved the Overton window and changed which sorts of opinions would earn you the horror of contemptuous looks and lowered status, and my negative model of OpenPhil is that they miraculously arrived at a set of opinions which would balance which sort of looks they got from a weighted set of people they cared about. So whatever happened after the ChatGPT Moment is no longer reflective of what I guess to be the organizational and cognitive processes underlying their earlier failure; and it's fair to ask about this because the earlier stuff was consequential. (Though it didn't move the needle as such; in retrospect and with benefit of hindsight, the needle started at "Dead" and stayed at "Dead" through everything MIRI or OpenPhil tried or failed at.)
While it is now possible to lose a lot of credit for having >30yr median timelines, it is no longer possible to earn significant credit for putting your timelines under 2055 because that is already what "the weighted average of facial expressions on people you care about" is telling you to believe and there are no big social penalties for believing it.
My view of the tragedy of OpenPhil is indeed that they were very earnest people trying to figure out what was legit, but ended up believing stuff like "biologically anchored estimates of AI timelines" that were facially absurd and wrong and ultimately self-serving, because the problem "end up with beliefs about AI timelines that aren't influenced by what plays well with our funders and friends" was hard and frankly out of their league and OpenPhil did not know that it was a hard problem or treat it with what I would consider seriousness.
If you'd like to view them as blameless on account of being earnest about it, that's between you and your own moral judgments. I don't particularly think we end up living through this if only we go around morally judging people enough, even correctly. But people ask me for my takes and I am giving a take that makes OpenPhil look bad and my rules do say that I ought to not just do all that behind their backs.
I suppose if you thought that nobody could possibly look bad if my account of them includes, "They were being very earnest in their error", then I wouldn't be obliged to give them a chance to respond to what I was saying about them. But I should prefer to have the chance to respond if somebody was saying that about me. Of course I am earnest, and when I err, it comes from a place of my having tried to be virtuous rather than viceful as best I understood virtue. What of it? There are higher things to aspire to in life besides earnest error.
What I'm pointing at there is that for strength/weight purposes, using big calcium nuclei to create stronger individual bonds in bone, is like making a steel beam stronger by putting more steel into it; the strength costs weight.
I'm not sure what's a truer analogy than static cling for hydrophobia as a force holding things together which the general audience has any experience with. Macroscopic experience of hydrophobia is, like, oil collecting on the surface of water, which isn't experienced as a binding force the way that static cling is.
I'm sort of skeptical that you could write something that works as science communication for a general audience, though lord knows I'm not necessarily succeeding either. The key valid ideas to be communicated are:
There exists a level above biology for molecular systems, greatly superior in terms of strength and energy density. This sets a lower bound on how a very smart and uncaring entity could kill you, which looks like it attacking you with micron-diameter robots, which looks like everyone on Earth falling over dead in the same second.
The designed micron-diameter thingies can easily kill you, where bacteria can't, because the designed thingies can more easily rip apart human cell membranes or white blood cells made of flimsier materials. They can do that because human cell membranes are held together by static cling, as are bacterial cells; whereas the ideal limits of what micron-sized engines can be put together are more like "diamond".
This design space isn't accessible to natural selection despite being physically possible, because evolutionary biology has an incredibly hard time designing systems like freely rotating wheels; for reasons that generalize to evolution not creating airborne cell-engines with solid covalently bonded shells and manipulator ports. My attempt to compress "Why?" down to something maybe overly pithy is "Because shallow energy gradients are more densely connected in the design space of simple mutations than deep energy gradients."
Now, instead of talking about human cell membranes being held together by static cling, I could talk about extremely thin metallic twisty-tie wires with some magnetized sections that help them fold up together into particular configurations in a barrel of magnetized ball bearings. Your suggestion above for science communication is that this is a great thing to mention, because it helps convey the following interesting truth: if we churn the ball bearings hard enough to unfold the twisty tie, it'll sometimes fold right back up into the same shape again once we stop churning!
This more complicated metaphor may legit add something to an explanation of organic chemistry. I don't disagree that it's cool, or important to organic chemistry proper.
From the perspective of explaining how you die when you confront an uncaring mind that thinks smarter and much faster than humanity, it doesn't add anything not already contained in "cell membranes are held together by static cling".
All of the difficulty here is having the sign of your impact be positive. It's very hard to end up neutral; eg, if your work is just nonsense, it's negative because it's a distraction and attention sink. And it's quite easy to end up negative, for example, if you exaggerate the impact of your work and provide more feed into the hopium ecosystem that desperately touts up any sign of progress.
When you mess with AI, whatever you do will outweigh any other impacts of your life, of course, obviously. It's having the sign end up positive that is the hard part.
I got a one-time gift of appreciated crypto, not through MIRI, part of whose purpose as I understood it was to give me enough of a savings backstop (having in previous years been not paid very much at all) that I would feel freer to speak my mind or change my mind should the need arise.
I have of course already changed MIRI's public mission sharply on two occasions, the first being when I realized in 2001 that alignment might need to be a thing, and said so to the primary financial supporter who'd previously supported MIRI (then SIAI) on the premise of charging straight ahead on AI capabilities; the second being in the early 2020s when I declared publicly that I did not think alignment technical work was going to complete in time and MIRI was mostly shifting over to warning the world of that rather than continuing to run workshops. Should I need to pivot a third time, history suggests that I would not be out of a job.
Okay; I guess I was confused by your question because I thought I'd said that in the main doc.
To repeat and with added explanation: Only opinions from before ChatGPT count.
This is because ChatGPT moved the Overton window and changed which sorts of opinions would earn you the horror of contemptuous looks and lowered status, and my negative model of OpenPhil is that they miraculously arrived at a set of opinions which would balance which sort of looks they got from a weighted set of people they cared about. So whatever happened after the ChatGPT Moment is no longer reflective of what I guess to be the organizational and cognitive processes underlying their earlier failure; and it's fair to ask about this because the earlier stuff was consequential. (Though it didn't move the needle as such; in retrospect and with benefit of hindsight, the needle started at "Dead" and stayed at "Dead" through everything MIRI or OpenPhil tried or failed at.)
While it is now possible to lose a lot of credit for having >30yr median timelines, it is no longer possible to earn significant credit for putting your timelines under 2055 because that is already what "the weighted average of facial expressions on people you care about" is telling you to believe and there are no big social penalties for believing it.
It would meet my criteria for being smarter than a potted plant, I suppose.
The harm's already been done.
My view of the tragedy of OpenPhil is indeed that they were very earnest people trying to figure out what was legit, but ended up believing stuff like "biologically anchored estimates of AI timelines" that were facially absurd and wrong and ultimately self-serving, because the problem "end up with beliefs about AI timelines that aren't influenced by what plays well with our funders and friends" was hard and frankly out of their league and OpenPhil did not know that it was a hard problem or treat it with what I would consider seriousness.
If you'd like to view them as blameless on account of being earnest about it, that's between you and your own moral judgments. I don't particularly think we end up living through this if only we go around morally judging people enough, even correctly. But people ask me for my takes and I am giving a take that makes OpenPhil look bad and my rules do say that I ought to not just do all that behind their backs.
I suppose if you thought that nobody could possibly look bad if my account of them includes, "They were being very earnest in their error", then I wouldn't be obliged to give them a chance to respond to what I was saying about them. But I should prefer to have the chance to respond if somebody was saying that about me. Of course I am earnest, and when I err, it comes from a place of my having tried to be virtuous rather than viceful as best I understood virtue. What of it? There are higher things to aspire to in life besides earnest error.
What I'm pointing at there is that for strength/weight purposes, using big calcium nuclei to create stronger individual bonds in bone, is like making a steel beam stronger by putting more steel into it; the strength costs weight.
Why is horn weaker than diamond?
I agree that "biology sticks to ionic bonds and static cling" was badly put because lignin, and I'll retire that one.
I'm not sure what's a truer analogy than static cling for hydrophobia as a force holding things together which the general audience has any experience with. Macroscopic experience of hydrophobia is, like, oil collecting on the surface of water, which isn't experienced as a binding force the way that static cling is.
I'm sort of skeptical that you could write something that works as science communication for a general audience, though lord knows I'm not necessarily succeeding either. The key valid ideas to be communicated are:
Now, instead of talking about human cell membranes being held together by static cling, I could talk about extremely thin metallic twisty-tie wires with some magnetized sections that help them fold up together into particular configurations in a barrel of magnetized ball bearings. Your suggestion above for science communication is that this is a great thing to mention, because it helps convey the following interesting truth: if we churn the ball bearings hard enough to unfold the twisty tie, it'll sometimes fold right back up into the same shape again once we stop churning!
This more complicated metaphor may legit add something to an explanation of organic chemistry. I don't disagree that it's cool, or important to organic chemistry proper.
From the perspective of explaining how you die when you confront an uncaring mind that thinks smarter and much faster than humanity, it doesn't add anything not already contained in "cell membranes are held together by static cling".