In this topic, you share a text relevant to EA, such as an article, essay, blog post, book or academic paper. I tell you three errors.
I’m offering to help EA by finding errors that people didn’t know about. Please only submit texts for which knowing errors would be valuable to you. I hope this will be useful and appreciated.
Details
If I post errors for your text, you must choose to debate one of the errors with me or choose not to debate. A one sentence reply explicitly opting out of debating is fine, but silence violates the game rules. Other feedback, such as which errors you agree or disagree with, is also welcome.
I only guarantee to do this for up to 5 submissions made within 5 days. First come, first serve. Limit 1 per person.
You must have already read the text in full yourself and like it a lot. (If you skipped reading notes or appendices, that’s fine, but state it.)
I must be able to find a free, electronic copy of the text. I can frequently find this for paywalled texts. If you already have a link or the file itself, please send it to me (DMs are fine).
If I can’t find three errors, I’ll say that. I don’t expect this to come up much. If it does, my expectation was wrong. I have two beliefs here. First, I’ll be able to find errors in texts that I disagree with. Second, people here are likely to share stuff I have disagreements with. To give a number, I predict finding errors for at least 80% of texts.
I expect prose texts over 1000 words long that say something reasonably substantial and complex. Otherwise I may aim to find fewer errors.
I’m not agreeing to read the whole text. My plan is to read enough to find three errors then stop. If something is addressed in a part I didn’t read, you can tell me and I’ll respond. I have experience replying based on partial reading and it’s rarely a problem.
I will only post errors that I consider important. If you consider an error unimportant, let me know and I’ll explain my perspective. You’re welcome to do that before either choosing an error to debate or choosing not to debate. You may want to state why you think it’s unimportant so I can address your reasoning, but I can explain importance regardless.
Your EA forum account must have been created in Sept 2022 or earlier.
Bonuses
Maybe this will inspire someone else to host the same game or a similar game.
This can serve as some examples of replying to the first error (well, first three).
Introduction
FYI, I disagree with the singularity hypothesis, but primarily due to epistemology, which isn't even discussed in this article.
Error One
There are many other reasons for drug research progress to slow down. The healthcare industry, as well as science in general (see e.g. the replication crisis), are really broken, and some of the problems are newer. Also maybe they're putting a bunch of work into updates to existing drugs instead of new drugs.
Similarly, decreasing crop yield growths (in other words, yields are still increasing but by lower percentages) could have many other causes. And also decreasing crop yields are a different thing than a decrease in the number of new agricultural ideas that researchers come up with – it's not even the right quantity to measure to make his point. It's a proxy for the actual thing his argument relies on, and he makes no attempt to consider how good or bad of a proxy it is, and I can easily think of some reasons it wouldn't be a very good proxy.
The comment about researchers not becoming lazy, poorly educated or overpaid is an unargued assertion.
So these are bad arguments which shouldn't convince us of the author's conclusion.
Error Two
Asserting something is unlikely isn't an argument. His followup is to bring up Moore's law potentially ending, not to give an actual argument.
As with the drug and agricultural research, his points are bad because singularity claims are not based on extrapolating patterns from current data, but rather on conceptual reasoning. He didn't even claim his opponents were doing that in the section formulating their position, and my pre-existing understanding of their views is they use conceptual arguments not extrapolating from existing data/patterns (there is no existing data about AGI to extrapolate from, so they use speculative arguments, which is OK).
Error Three
You can't just assume that AGIs will be anything like current software including "AI" software like AlphaGo. You have to consider what an AGI would be like before you can even know if it'd be especially good at this or not. If the goal with AGI is in some sense to make a machine with human-like thinking, then maybe it will end up with some of the weaknesses of humans too. You can't just assume it won't. You have to envision what an AGI would be like, or what many different things it might be like that would work (narrow it down to various categories and rule some things out) before you consider the traits it'd have.
Put another way, in MIRI's conception, wouldn't mind design space include both AGIs that are good or bad at this particular category of task?
Error Four
This is wrong due to "at once" at the end. It'd be fine without that. You could speed up up 9 out of 10 parts, then speed up the 10th part a minute later. You don't have to speed everything up at once. I know it's just two extra words but it doesn't make sense when you stop and think about it, so I think it's important. How did it seem to make sense to the author? What was he thinking? What process created this error? This is the kind of error that's good to post mortem. (It doesn't look like any sort of typo; I think it's actually based on some sort of thought process about the topic.)
Error Five
Section 3.2 doesn't even try to consider any specific type of research an AGI would be doing and claim that good ideas would get harder to find for that and thereby slow down singularity-relevant progress.
Similarly, section 3.3 doesn't try to propose a specific bottleneck and explain how it'd get in the way of the singularity. He does bring up one specific type of algorithm – search – but doesn't say why search speed would be a constraint on reaching the singularity. Whether exponential search speed progress is needed depends on specific models of how the hardware and/or software are improving and what they're doing.
There's also a general lack of acknowledgement of, or engagement with, counter-arguments that I can easily imagine pro-singularity people making (e.g. responding to the good ideas getting harder to find point by saying some stuff about mind design space containing plenty of minds that are powerful enough for a singularity with a discontinuity, even if progress slows down later as it approaches some fundamental limits). Similarly, maybe there is something super powerful in mind design space that doesn't rely on super fast search. Whether there is, or not, seems hard to analyze, but this paper doesn't even try. (The way I'd approach it myself is indirectly via epistemology first.)
Error Six
Section 2 mixes Formulating the singularity hypothesis (the section title) with other activities. This is confusing and biasing, because we don't get to read about what the singularity hypothesis is without the author's objections and dislikes mixed in. The section is also vague on some key points (mentioned in my screen recording) such as what an order of magnitude of intelligence is.
Examples:
Here he's mixing explaining the other side's view with setting it up to attack it (as requiring a super high evidential burden due to such strong claims). He's not talking from the other side's perspective, trying to present it how they would present it (positively); he's instead focusing on highlighting traits he dislikes.
This isn't formulating the singularity hypothesis. It's about ways of opposing it.
Again this doesn't fit the section it's in.
Padding
Section 3 opens with some restatements of material from section 2 which was also in the introduction some. And look at this repetitiveness (my bolds):
Near the bottom of page 7 begins section 3.2:
Below that we read:
Page 8 near the top:
Later in that paragraph:
Also, page 11:
Page 17
Amount Read
I read to the end of section 3.3 then briefly skimmed the rest.
Screen Recording
I recorded my screen and made verbal comments while writing this:
https://www.youtube.com/watch?v=T1Wu-086frA