E

EliezerYudkowsky

3687 karmaJoined

Comments
98

Okay; I guess I was confused by your question because I thought I'd said that in the main doc.

To repeat and with added explanation:  Only opinions from before ChatGPT count.

This is because ChatGPT moved the Overton window and changed which sorts of opinions would earn you the horror of contemptuous looks and lowered status, and my negative model of OpenPhil is that they miraculously arrived at a set of opinions which would balance which sort of looks they got from a weighted set of people they cared about.  So whatever happened after the ChatGPT Moment is no longer reflective of what I guess to be the organizational and cognitive processes underlying their earlier failure; and it's fair to ask about this because the earlier stuff was consequential.  (Though it didn't move the needle as such; in retrospect and with benefit of hindsight, the needle started at "Dead" and stayed at "Dead" through everything MIRI or OpenPhil tried or failed at.)

While it is now possible to lose a lot of credit for having >30yr median timelines, it is no longer possible to earn significant credit for putting your timelines under 2055 because that is already what "the weighted average of facial expressions on people you care about" is telling you to believe and there are no big social penalties for believing it.

My view of the tragedy of OpenPhil is indeed that they were very earnest people trying to figure out what was legit, but ended up believing stuff like "biologically anchored estimates of AI timelines" that were facially absurd and wrong and ultimately self-serving, because the problem "end up with beliefs about AI timelines that aren't influenced by what plays well with our funders and friends" was hard and frankly out of their league and OpenPhil did not know that it was a hard problem or treat it with what I would consider seriousness.

If you'd like to view them as blameless on account of being earnest about it, that's between you and your own moral judgments.  I don't particularly think we end up living through this if only we go around morally judging people enough, even correctly.  But people ask me for my takes and I am giving a take that makes OpenPhil look bad and my rules do say that I ought to not just do all that behind their backs.

I suppose if you thought that nobody could possibly look bad if my account of them includes, "They were being very earnest in their error", then I wouldn't be obliged to give them a chance to respond to what I was saying about them.  But I should prefer to have the chance to respond if somebody was saying that about me.  Of course I am earnest, and when I err, it comes from a place of my having tried to be virtuous rather than viceful as best I understood virtue.  What of it?  There are higher things to aspire to in life besides earnest error.

What I'm pointing at there is that for strength/weight purposes, using big calcium nuclei to create stronger individual bonds in bone, is like making a steel beam stronger by putting more steel into it; the strength costs weight.

I agree that "biology sticks to ionic bonds and static cling" was badly put because lignin, and I'll retire that one.

I'm not sure what's a truer analogy than static cling for hydrophobia as a force holding things together which the general audience has any experience with.  Macroscopic experience of hydrophobia is, like, oil collecting on the surface of water, which isn't experienced as a binding force the way that static cling is.

I'm sort of skeptical that you could write something that works as science communication for a general audience, though lord knows I'm not necessarily succeeding either.  The key valid ideas to be communicated are:

  • There exists a level above biology for molecular systems, greatly superior in terms of strength and energy density.  This sets a lower bound on how a very smart and uncaring entity could kill you, which looks like it attacking you with micron-diameter robots, which looks like everyone on Earth falling over dead in the same second.
  • The designed micron-diameter thingies can easily kill you, where bacteria can't, because the designed thingies can more easily rip apart human cell membranes or white blood cells made of flimsier materials.  They can do that because human cell membranes are held together by static cling, as are bacterial cells; whereas the ideal limits of what micron-sized engines can be put together are more like "diamond".
  • This design space isn't accessible to natural selection despite being physically possible, because evolutionary biology has an incredibly hard time designing systems like freely rotating wheels; for reasons that generalize to evolution not creating airborne cell-engines with solid covalently bonded shells and manipulator ports.  My attempt to compress "Why?" down to something maybe overly pithy is "Because shallow energy gradients are more densely connected in the design space of simple mutations than deep energy gradients."

Now, instead of talking about human cell membranes being held together by static cling, I could talk about extremely thin metallic twisty-tie wires with some magnetized sections that help them fold up together into particular configurations in a barrel of magnetized ball bearings.  Your suggestion above for science communication is that this is a great thing to mention, because it helps convey the following interesting truth: if we churn the ball bearings hard enough to unfold the twisty tie, it'll sometimes fold right back up into the same shape again once we stop churning!

This more complicated metaphor may legit add something to an explanation of organic chemistry.  I don't disagree that it's cool, or important to organic chemistry proper.

From the perspective of explaining how you die when you confront an uncaring mind that thinks smarter and much faster than humanity, it doesn't add anything not already contained in "cell membranes are held together by static cling".

Why is flesh weaker than diamond?  Diamond is made of carbon-carbon bonds.  Proteins also have some carbon-carbon bonds!  So why should a diamond blade be able to cut skin?

I reply:  Because the strength of the material is determined by its weakest link, not its strongest link.  A structure of steel beams held together at the vertices by Scotch tape (and lacking other clever arrangements of mechanical advantage) has the strength of Scotch tape rather than the strength of steel.

Or:  Even when the load-bearing forces holding large molecular systems together are locally covalent bonds, as in lignin (what makes wood strong), if you've got larger molecules only held together by covalent bonds at interspersed points along their edges, that's like having 10cm-diameter steel beams held together by 1cm welds.  Again, barring other clever arrangements of mechanical advantage, that structure has the strength of 1cm of steel rather than 10cm of steel.

Bone is stronger than wood; it runs on a relatively stronger structure of ionic bonds, which are no locally weaker than carbon bonds in terms of attojoules of potential energy per bond.  Bone is weaker than diamond, then, because... why?

Well, partially, IIUC, because calcium atoms are heavier than carbon atoms.  So even if per-bond the ionic forces are strong, some of that is lost in the price you pay for including heavier atoms whose nuclei have more protons that are able to exert the stronger electrical forces making up that stronger bond.

But mainly, bone is so much weaker than diamond (on my understanding) because the carbon bonds in diamond have a regular crystal structure that locks the carbon atoms into relative angles, and in a solid diamond this crystal structure is tesselated globally.  Hydroxyapatite (the crystal part of bone) also tesselates in an energetically favorable configuration; but (I could be wrong about this) it doesn't have the same local resistance to local deformation; and also, the actual hydroxyapatite crystal is assembled by other tissues that layer the ionic components into place, which means that a larger structure of bone is full of fault lines.  Bone cleaves along the weaker fault line, not at its strongest point.

But then, why don't diamond bones exist already?  Not just for the added strength; why make the organism look for calcium and phosphorus instead of just carbon?

The search process of evolutionary biology is not the search of engineering; natural selection can only access designs via pathways of incremental mutations that are locally advantageous, not intelligently designed simultaneous changes that compensate for each other.  There were, last time I checked, only three known cases where evolutionary biology invented the freely rotating wheel.  Two of those known cases are ATP synthase and the bacterial flagellum, which demonstrates that freely rotating wheels are in fact incredibly useful in biology, and are conserved when biology stumbles across them after a few hundred million years of search.  But there's no use for a freely rotating wheel without a bearing and there's no use for a bearing without a freely rotating wheel, and a simultaneous dependency like that is a huge obstacle to biology, even though it's a hardly noticeable obstacle to intelligent engineering.

The entire human body, faced with a strong impact like being gored by a rhinocerous horn, will fail at its weakest point, not its strongest point.  How much evolutionary advantage is there to stronger bone, if what fails first is torn muscle?  How much advantage is there to an impact-resistant kidney, if most fights that destroy a kidney will kill you anyways?  Evolution is not the sort of optimizer that says, "Okay, let's design an entire stronger body."  (Analogously, the collection of faults that add up to "old age" is large enough that a little more age resistance in one place is not much of an advantage if other aging systems or outward accidents will soon kill you anyways.)

I don't even think we have much of a reason to believe that it'd be physically (rather than informationally) difficult to have a set of enzymes that synthesize diamond.  It could just require 3 things to go right simultaneously, and so be much much harder to stumble across than tossing more hydroxyapatite to lock into place in a bone crystal.  And then even if somehow evolution hit on the right set of 3 simultaneous mutations, sometime over the history of Earth, the resulting little isolated chunk of diamond probably would not be somewhere in the phenotype that had previously constituted the weakest point in a mechanical system that frequently failed.  If evolution has huge difficulty inventing wheels, why expect that it could build diamond chainmail, even assuming that diamond chainmail is physically possible and could be useful to an organism that had it?

Talking to the general public is hard.  The first concept I'm trying to convey to them is that there's an underlying physical, mechanical reason that flesh is weaker than diamond; and that this reason isn't that things animated by vitalic spirit, elan vital, can self-heal and self-reproduce at the cost of being weaker than the cold steel making up lifeless machines, as is the price of magic imposed by the universe to maintain game balance.  This is a very natural way for humans to think; and the thing I am trying to come in and do is say, "Actually, no, it's not a mystical balance, it's that diamond is held together by bonds that are hundreds of kJ/mol; and the mechanical strength of proteins is determined by forces a hundred times as weak as that, the part where proteins fold up like spaghetti held together by static cling."

There is then a deeper story that's even harder to explain, about why evolution doesn't build freely rotating wheels or diamond chainmail; why evolutionary design doesn't find the physically possible stronger systems.  But first you need to give people a mechanical intuition for why, in a very rough intuitive sense, it is physically possible to have stuff that moves and lives and self-repairs but is strong like diamond instead of flesh, without this violating a mystical balance where the price of vitalic animation is lower material strength.

And that mechanical intuition is:  Deep down is a bunch of stuff that, if you could see videos of it, would look more like tiny machines than like magic, though they would not look like familiar machines (very few freely rotating wheels).  Then why aren't these machines strong like human machines of steel are strong?  Because iron atoms are stronger than carbon atoms?  Actually no, diamond is made of carbon and that's still quite strong.  The reason is that these tiny systems of machinery are held together (at the weakest joints, not the strongest joints!) by static cling.

And then the deeper question:  Why does evolution build that way?  And the deeper answer:  Because everything evolution builds is arrived at as an error, a mutation, from something else that it builds.  Very tight bonds fold up along very deterministic pathways.  So (in the average case, not every case) the neighborhood of functionally similar designs is densely connected along shallow energy gradients and sparsely connected along deep energy gradients.  Intelligence can leap long distances through that design space using coordinated changes, but evolutionary exploration usually cannot.

And I do try to explain that too.  But it is legitimately more abstract and harder to understand.  So I lead with the idea that proteins are held together by static cling.  This is, I think, validly the first fact you lead with if the audience does not already know it, and just has no clue why anyone could possibly possibly think that there might even be machinery that does what bacterial machinery does but better.  The typical audience is not starting out with the intuition that one would naively think that of course you could put together stronger molecular machinery, given the physics of stronger bonds, and then we debate whether (as I believe) the naive intuition is actually just valid and correct; they don't understand what the naive intuition is about, and that's the first thing to convey.

If somebody then says, "How can you be so ignorant of chemistry?  Some atoms in protein are held together by covalent bonds, not by static cling!  There's even eg sulfur bonds whereby some parts of the folded-spaghetti systems end up glued together with real glue!" then this does not validly address the original point because: the underlying point about why flesh is more easily cleaved than diamond, is about the weakest points of flesh rather than the strongest points in flesh, because that's what determines the mechanical strength of the larger system.

I think there is an important way of looking at questions like these where, at the final end, you ask yourself, "Okay, but does my argument prove that flesh is in fact as strong as diamond?  Why isn't flesh as strong as diamond, then, if I've refuted the original argument for why it isn't?" and this is the question that leads you to realize that some local strong covalent bonds don't matter to the argument if those bonds aren't the parts that break under load.

My main moral qualm about using the Argument From Folded Spaghetti Held Together By Static Cling as an intuition pump is that the local ionic bonds in bone are legitimately as strong per-bond as the C-C bonds in diamond, and the reason that bone is weaker than diamond is (iiuc) actually more about irregularity, fault lines, and resistance to local deformation than about kJ/mol of the underlying bonds.  If somebody says "Okay, fine, you've validly explained why flesh is weaker than diamond, but why is bone weaker than diamond?" I have to reply "Valid, iiuc that's legit more about irregularity and fault lines and interlaced weaker superstructure and local deformation resistance of the bonds, rather than the raw potential energy deltas of the load-bearing welds."

I broadly endorse this reply and have mostly shifted to trying to talk about "covalently bonded" bacteria, since using the term "diamondoid" (tightly covalently bonded CHON) causes people to panic about the lack of currently known mechanosynthesis pathways for tetrahedral carbon lattices.

Load more