Engineer working on next-gen satellite navigation at Xona Space Systems. I write about rationalist and longtermist topics at jacksonw.xyz
See my response to kokotajlod to maybe get a better picture of where I am coming from and how I am thinking about the contest."Directly conflicts with the geopolitical requirements." -- How would asking the AGI to take it slow conflict with the geopolitical requirements? Imagine that I invent a perfectly aligned superintelligence tomorrow in my spare time, and I say to it, "Okay AGI, I don't want things to feel too crazy, so for starters, how about you give humanity 15% GDP growth for the next 30 years? (Perhaps by leaking designs for new technologies discreetly online.) And make sure to use your super-persuasion to manipulate public sentiment a bit so that nobody gets into any big wars." That would be 5x the current rate of worldwide economic growth, which would probably feel like "transforming the economy sector by sector" to most normal people. I think that world would perfectly satisfy the contest rules. The only problems I can see are:
I agree with you that there's a spectrum of different things that can be meant by "honesty", sliding from "technically accurate statements which fail to convey the general impression" to "correctly conveying the general impression but giving vague or misleading statements", and that in some cases the thing we're trying to describe is so strange that no matter where we go along that continuum it'll feel like lying because the description will be misleading in one way or the other. That's a problem with full Eutopia but I don't think it's a problem with the 2045 story, where we're being challenged not to describe the indescribable but to find the most plausible path towards a goal (a familiar but peaceful and prosperous world) which, although very desirable to many people, doesn't seem very likely if AGI is involved.I think the biggest risk of dishonesty for this contest, is if the TRUE most-plausible path to a peaceful & prosperous 2045 (even one that satisfies all the geopolitical conditions) still lies outside the Overton Window of what FLI is willing to publish, so instead people choose to write about less plausible paths that probably won't work. (See my "cabal of secret geniuses runs the show from behind the scenes" versus "USA/China/EU come together to govern AI for all mankind" example in my comment to kokotajlod -- if the cabal of secret geniuses path is what we should objectively be aiming for but FLI will only publish the latter story, that would be unfortunate.)
Maybe you think the FLI contest is immoral for exactly this reason -- because the TRUE most-plausible path to a good future doesn't/couldn't go through . Yudkowsky has said a few things to this effect, about how no truly Pivotal Action (something your aligned AGI could do to prevent future unaligned AGIs from destroying the world) fits inside the overton window, and he just uses "have the AGI create a nanosystem to melt all the world's GPUs" (which I guess he sees as being an incomplete solution) as a politically palatable illustrative example. I'm not sure about this question and I'm open to being won over.
To clarify, I'm not affiliated with FLI, so I'm not the one imposing the constraints, they are. I'm just defending them, because the contest rules seem reasonable enough to me. Here are a couple of thoughts:
Anyways, on to the more important issue of this actual contest, the 2045 AGI story, and its oddly-specific political requirements:
You say: "[This scenario seems so unrealistic that I can only imagine it happening if we first align AGI and then request that it give us a slow ride even though it's capable of going faster.] ...Would you accept interpretations such as this?"
I'm not FLI so it's not my job to say which interpretations are acceptable, but I'd say you're already doing exactly the work FLI was looking for! I agree that this scenario is one of the most plausible ways that civilization might end up fulfilling the contest conditions. Here are some other possibilities:
See my response to Czynski for more assorted thoughts, although I've written so much now that perhaps I could have entered the contest myself by now if I had been writing stories instead of comments! :PEdited to add that alas, I only just now saw your other comment about "So in order to describe a good future, people will fiddle with the knobs of those important variables so that they are on their conducive-to-good settings rather than their most probable settings. ". This strikes me as a fair criticism of the contest. (For one, it will bias people towards handwaving over the alignment problem by saying "it turned out to be surprisingly easy".) I don't think that's devastating for the contest, since I think there's a lot of value in just trying to envision what an agreeable good outcome for humanity looks like. But definitely a fair critique that lines up with the stuff I was saying above -- basically, there are both pros and cons to putting $100K of optimization pressure behind getting people to figure out the most plausible optimistic outcome under a set of constraints. (Maybe FLI should run another contest encouraging people to do more Yudkowsky-style brainstorming of how everything could go horribly wrong before we even realize what we were dealing with, just to even things out!)
Hi! Separately from this current arbitrage opportunity, does the new NY law represent any kind of meaningful advance for the legality of prediction markets?Also, if there are any rationalist sports fans out there with hot takes (or perhaps cold ones) on how I should bet, I'd be open to taking your advice. After all, if I sign up for this, I will have to end up placing some random sports bets one way or another. Might as well try to grab some of that mythical rationalist alpha while I'm at it.
The contest is only about describing 2045, not necessarily a radically alien far-future "Eutopia" end state of human civilization. If humans totally solve alignment, we'd probably ask our AGI to take us to Eutopia slowly, allowing us to savor the improvement and adjust to the changes along the way, rather than leaping all the way to the destination in one terrifying lurch. So I'm thinking there are probably some good ways to answer this prompt.
But let's engage with the harder question of describing a full Eutopia. If Eutopia is truly good, then surely there must be honest ways of describing it that express why it is good and desirable, even if Eutopia is also scary. Otherwise you'd be left with three options that all seem immoral:
It's impossible to imagine infinity, but if you're trying to explain how big infinity is, surely it's better to say "it's like the number of stars in the night sky", or "it's like the number of drops of water in the ocean", than to say "it's like the number of apples you can fit in a bucket". Similarly, the closest possible description of the indescribable Eutopia must be something that sounds basically good (even if it is clearly also a little unfamiliar), because the fundamental idea of Eutopia is that it's desirable. I don't think that's lying, anymore than trying to describe other indescribable things as well as you can is lying.Yudkowsky's own essay "Eutopia is Scary" was part of a larger "Fun Theory" sequence about attempting to describe utopias. He mostly described them in a positive light, with the "Eutopia is Scary" article serving as an important, but secondary, honesty-enhancing caveat: "these worlds will be a lot of fun, but keep in mind they'll also be a little strange".
This post was helpful to me in understanding what I should aim to accomplish with my own personal donations. I expect that many other EAs feel similarly -- donating is an important part of being an EA for many people, but the question of how to maximize impact as a small-scale individual donor is a complex puzzle when you consider the actions of other donors and the community as a whole. This post is a clear, early articulation of key themes that show up in the continual debate and discussion that surround real-world individual donation decisions. I think it has stood the test of time and could easily go on to help more EAs figure out their donations in the future, justifying its inclusion in the decadal review.
This is going to be a quick review since I there has been plenty of discussion of this post and people understand it well. But this post was very influential for me personally, and helped communicate yet another aspect of the key problem with AI risk -- the fact that its so unprecedented, which makes it hard to test and iterate solutions, hard to raise awareness and get agreement about the nature of the problem, and hard to know how much time we have left to prepare.AI is simply one of the biggest worries among longtermist EAs, and this essay does a good job describing a social dynamic unique to the space of AI risk that makes dealing with the risk harder. For this reason it would be a fine inclusion in the decadal review.
I think this research into x-risk & economic growth is a good contribution to patient longtermism. I also think that integrating thoughts on economic growth more deeply into EA holds a lot of promise -- maybe models like this one could someday form a kind of "medium-termist" bridge between different cause areas, creating a common prioritization framework. For both of these reasons I think this post is worth of inclusion in the decadal review.The question of whether to be for or against economic growth in general is perhaps not the number-one most pressing dilemma in EA (since everyone agrees that differential technology development into x-risk-reducing areas is very important), but it is surely up there, since it's such a big-picture question that affects so many decisions. Other than X-risk concerns, economic growth obviously looks attractive -- both in the developing world where it's a great way to make the world's poorest people more prosperous, or in the first world where the causes championed by "progress studies" promise to create a prosperous and more dynamic society where people can live better lives. But of course, by longtermist lights, how fast we get to the future is less important than making sure we get there at all. So, in the end, what to do about influencing economic growth? Leopold's work is probably just a starting point for this huge and perhaps unanswerable set of questions. But it's a good start -- I'm not sure if I want more economic growth or not, but I definitely want more posts like these tackling the question.For the decadal review, rather than the literal text of this post (which merely refers to the pdf) or the comprehensive 100-page pdf itself, I'd suggest including Leopold's "Works in Progress" article summarizing his research.
There are many reasons why I think this post is good:
As the Creative Writing Contest noted, Singer's drowing-child thought experiment "probably did more to launch the EA movement than any other piece of writing". The key elements of the story have spread far and wide -- when I was in high school in 2009, an English teacher of mine related the story to my class as part of a group discussion, years before I had ever heard of Effective Altruism or anything related to it. Should this post be included in the decadal review? Certainly, its importance is undisputed. If anything, Singer's essay might be too well-known to merit inclusion in a decadal review, since its basic logic (that suffering far away still matters, and that suffering far away can sometimes be averted very cheaply) is essentially the starting-point through which almost all new EAs are introduced to the movement. It also might fail due to the technicality that the story was originally written in 1997, despite being reposted on our dear Forum in 2014.Certainly there have been criticisms of the story, such as Yudkowsky's here. It seems a bit of a bait and switch to have the story be about diving into a pool (which only takes a few minutes and at most ruins a nice set of clothes), and then Singer says that we are all in the situation described due to the existence of charities like Against Malaria Foundation who can save a life for several thousand dollars (which even for most rich-world citizens is more like the hard-earned savings from several months' labor). But that is just nitpicking plot details -- the fundamentals of the story are sound.
I see Gwern's/Aaron's post about The Narrowing Circle as part of an important thread in EA devoted to understanding the causes of moral change. By probing the limits of the "expanding circle" idea for counterexamples, perhaps we can understand it better.Effective altruism is popular among moral philosophers, and EAs are often seeking to expand people's "moral circle of concern" towards currently neglected classes of beings, like nonhuman animals or potential future generations of mankind. This is a laudable goal (and one which I share), but it's important that the movement does not get carried away with the assumption that such a cultural shift is inevitable. The phenomenon of the expanding circle must be caused by something, and those causes are probably driven by material conditions that could change or reverse in the future.As I see it, the strongest part of the argument for a "narrowing circle" is the "Ancestors" and "Descendants" sections. It seems plausible to me that preindustrial "farmer" culture placed nigh-obsessive emphasis on pleasing the wishes of your ancestors and securing a promising future for your descendants. (I suspect this is probably because, in a world where income came from farming the land rather than hunting/gathering or performing skilled industrial-age work for wages, inheritance of farmland from one generation to the next becomes crucially important.) Much of the modern world seems to have essentially abandoned the idea that we should place much weight on the values of our ancestors, which should be concerning to longtermists since valuing the lives of ancestors seems very close to valuing the lives of unborn generations (see for instance Chesterton's quote about how "tradition is the democracy of the dead").The idea that concern for descendants has also decreased is certainly a worry worth investigating -- perhaps a logical place to start would be by investigating how much the recent worldwide decline in fertility rates really reflects a decreased desire for children. A drop in respect for ancestors might also directly cause a drop in concern for descendants -- it might be logical to disregard the lives of future generations if we assume that they (just like us) will ignore the wishes of their ancestors!Anyways, here are some other pieces that seem relevant to the thread of "investigating what drives moral change":- AppliedDivinityStudies arguing that moral philosophy is not what actually drives moral progress.- A lot of Slate Star Codex / Astral Codex Ten is about understanding cultural changes. Here for instance is a dialogue about shifting moral foundations, expanding circles, and what that might tell us about how things will continue to shift in the future.Finally, I think that investigating the "expanding circle" is doubly important because it's not just an assumption of a couple people within the nascent EA movement... it's very similar to one of the core legitimizing stories that are held up to justify mainstream liberal democracy! I am talking about the whole civil-rights story that "the moral arc of the universe bends towards justice", that democracy is good because it has lead an expanding-circle-style transition towards increasingly recognizing the civil rights of women, minorities, etc. I think this story is true, but I don't know exactly why and I don't think the trend is guaranteed to continue. (Was democracy itself the cause, or merely another symptom of a larger force like the Industrial Revolution?)For all these reasons, I think this is a good post worthy of inclusion in the decadal review.