Even after GPT-4, AI researchers forecasted a 50% chance of AGI by 2047 or 2116, depending how you define AGI

Good question. To know for sure, you would have to collect data on the rate of at-fault collisions for human drivers in specifically just those geofenced areas, which would be onerous. I don't particularly blame Waymo for not collecting this data, but it has to be said this is a methodological problem. 

There's also been the concern raised for a long time that Waymos may cause collisions that, legally, they aren't at fault for. So, if Waymos are prone to suddenly breaking due to harmless obstacles like plastic bags or simply because of mistakes made by the perception system, then they could get rear-ended a lot, but rear-ending is always the fault of the car in the rear as a matter of law. Legally, this would count as completely safe, but common sense tells us that if a technology is statistically causing a large number of collisions, that's actually a safety concern.

You did mention serious accidents specifically. I should look at the safety paper that was published by Waymo this year to see how they address this. This is the kind of disclosure of internal safety data that self-driving car companies have historically been extremely reluctant to make and that I would have killed to get my hands on back when I was writing a lot about self-driving cars. But the general difficulty can be seen in this report published by RAND Corporation in 2016. More serious accidents (those causing injuries or deaths) are much less common than minor collisions so, statistically, you need a lot more data before you can be confident about their actual frequency. The Waymo safety paper says it's looking at data from 57 million miles. According to the RAND report, that wouldn't be quite enough data to establish a lower collision rate than human drivers or even half as much as you'd need to establish a lower injury rate, but I guess maybe it could if the frequency were much, much less in the observed data. (However, you'd still have the bias introduced by comparing data from driving in geofenced areas to data collected about humans driving everywhere.) 

I am most curious of all to see what independent experts make of Waymo's paper, since of course Waymo will try to make itself look as good as possible and will omit anything too inconvenient. 

What I want to emphasize more than anything is the distinction between safety and competence or overall performance. Particularly in the context of forecasting widespread deployment of robotaxis or using autonomous driving as a point of reference for AI generally or for AGI. A car that just stays parked all the time is 100% safe. With current technology, you could most likely get an autonomous car to drive around a closed track indefinitely and be almost 100% safe. Same for a car just circling the block in one of those fake cities in California used for testing. Safety is not the same as competence for overall performance. 

Intuitively, a self-driving car being safer than a human driver evokes the idea that the car can drive anywhere and do anything, all its own, and it gets into collisions less often. People point to safety claims from Waymo and say self-driving is a solved problem — we already have a superhuman AI driver. But all that Waymo is really claiming is that the rate its vehicles get into collisions and cause injuries is less than the average rate than human drivers. I don't think Waymo would make the stronger version of the claim because they know that's not true and they would probably worry about being sued by investors or getting punished by the SEC. (Well, maybe not under this administration. Is there still an SEC?) 

If safety were the same as overall performance or competence, there would be no need for geofencing and there would be no need for teleoperation or remote assistance. You could just let driveless vehicles loose anywhere and everywhere with no human in the loop. 

A good analogy for Waymo is "chimeras" in chess, which combined chess-playing AIs and human players. It's a system that combines AI and human intelligence, not a pure AI system. This is how Waymo describes how its remote assistance works in a May 2024 blog post:

Much like phone-a-friend, when the Waymo vehicle encounters a particular situation on the road, the autonomous driver can reach out to a human fleet response agent for additional information to contextualize its environment. The Waymo Driver does not rely solely on the inputs it receives from the fleet response agent and it is in control of the vehicle at all times. ...fleet response can view real-time feeds from the vehicle’s exterior cameras and a 3D graphical representation of what the car perceives around it. ... Fleet response and the Waymo Driver primarily communicate through questions and answers. For example, suppose a Waymo AV approaches a construction site with an atypical cone configuration indicating a lane shift or close. In that case, the Waymo Driver might contact a fleet response agent to confirm which lane the cones intend to close. ... Fleet response can influence the Waymo Driver's path, whether indirectly through indicating lane closures, explicitly requesting the AV use a particular lane, or, in the most complex scenarios, explicitly proposing a path for the vehicle to consider.

So, the humans are not remote controlling the car like they're playing a racing game (à la Grand Turismo), it's more like they're playing The Sims and saying "go here", "do this". Although it sounds like they can also even draw a specific path for the car to follow. (To visualize what drawing a path looks like, look at the orange ribbon in this video. That's what the car is "predicting" or planning as its path.)

Clearly, this is not the same thing as a car that actually knows how to drive by itself. What's worse, the human's role in the system is invisible, so we don't know what's AI and what's a Mechanical Turk. 

Looking at Tesla is probably more informative to see the state of the art. Teslas have cheaper sensors (i.e. no lidar) but the fleet is 1,000x larger, so Tesla has a 1,000x larger source of data to sample from and train on. When you watch the vlogs of Teslas in semi-autonomous mode, you see when the human intervenes. If you watch a vlog of a Waymo trip, you will never be able to tell when a human is intervening remotely or not. 

There's also what I mentioned before, which is that strict geofencing allows Waymo to special case the whole thing, down to each individual bush next to the road. This is a quote from a 2018 article about Cruise, not Waymo, but the same concepts apply:

“Cruise cars frequently swerve and hesitate,” Efrati reports. “They sometimes slow down or stop if they see a bush on the side of a street or a lane-dividing pole, mistaking it for an object in their path.” In one case, Efrati says, Cruise employees trimmed a bush ahead of a demonstration for journalists to make sure the car wouldn’t swerve while driving past it.

Cruise employees have the option of riding in Cruise vehicles as they travel around San Francisco, but there’s a big downside to doing so: self-driving car rides are often slower than a ride in a human vehicle—sometimes as much as 10 to 20 minutes slower. A big reason for that: “some San Francisco intersections and streets are ‘blacklisted,’ in some cases temporarily, and the cars must take circuitous routes around them.”

An intersection might be blacklisted because its traffic light is too faint, because it has a complex roundabout, or because it requires a difficult lane merge.

Like other companies, Waymo uses HD maps which require a lot of human annotation and constant updating. So, instead of sending someone out to trim the bush, an employee might annotate the bush as something that causes problems that the car needs to give a wide berth when it's in that portion of that lane, or if it's really problematic they might just mark that street off as somewhere the Waymos don't go. 

There's also been this big debate and conversation in autonomous driving tech for many years about hand-coded behaviours vs. machine learned behaviours. Both have well-known issues. Hand-coded behaviours are extremely brittle. It's hard to imagine even a million engineers at a million computers typing enough rules to handle every scenario a car could ever encounter in the world. Machine learned behaviours are brittle in their own way, i.e., neural networks can go crazy when they encounter something weird or novel that doesn't match their training dataset. So, there's been a lot of discussion about the strengths and weaknesses of each approach, and proposals for the right combination or hybridization of them.

This is more on the level of rumour or speculation than reliable knowledge, but what I've heard conjectured is that Waymo engineers solve one problem at a time as they come up in their geofenced areas. In principle, these hand-coded rules might be general purpose, e.g., whenever a car encounters a bush, give it a wide berth. But then you don't know if these rules will generalize well outside of the geofences areas. What if that's a good policy for bushes in San Francisco or Austin, but another city has bushes planted on the median dividing the street? Then you might have the Waymo getting uncomfortable close to parked cars or pedestrians on the sidewalk. Just a made-up example. The overall idea is that if, statistically, every self-driving car has its own personal engineer writing code to solve the problems it encounters on its daily routes, then maybe that will work for very limited deployments, but it won't scale to 280 million vehicles. 

This is a very long comment in response to a very short question. But I think it's worth getting into the weeds on this. If the whole fate of the world depends on whether AGI is imminent or not, then self-driving cars are an important point of comparison to get right. Some people claim that self-driving is solved and use that as evidence to argue for near-term AGI. But self-driving isn't solved and, actually, the fact that it isn't solved raises inconvenient questions for the hypothesis that AGI will be invented soon. 

Tesla's fleet does something like 800 years of driving per year, so it's hard to argue the problem is just a lack of data rather than something more fundamental. And is not data inefficiency itself a fundamental problem with the current paradigm of AI? You can try to argue that computer vision and navigating a body or robot around a physical environment is something entirely different than the kind of cognitive capabilities that would constitute AGI, but then are we really saying that AGI won't be able to see, walk, or drive? Really? 

I think Waymo, Cruise, et al. are also a good lesson about how technology companies can put up an appearance of inevitable progress racing toward a transformative endpoint, but if you do a little detective work the story falls apart. I was a bit scandalized when François Chollet said that, like how Waymo does the special casing with HD maps and probably with hand-coded rules as well, OpenAI and other LLM companies use a workforce of tens of thousands of full-time contractors (across the whole LLM industry) to apply special case fixes to mistakes LLMs make. This gives the appearance of increasing LLM intelligence, but it's just a Mechanical Turk!