Specifically, an agent that has a few things true about it, like "it won't trade things in a circle so that it will end up losing something and gaining nothing" will have a goal that can be described with a utility function.

Something is wrong here, because I fit the description of an "AGI", and yet I do not have a utility function. Within that theorem something is being smuggled in that is not necessary for general intelligence. 

Hey Yonatan,

first, excuse my spelling your name incorrectly originally, I fixed it now.

Thank you for your encouragement with funding. As it happens, we did apply for funding from several sources and are waiting for their response.

Regarding Rob Miles' videos on satisficing:

One potential misunderstanding relates to the question of with what probability the agent is required to reach a certain goal. If I understand him correctly, he assumes satisficing needs to imply maximizing the probability that some constraint is met, which would still constitute a form of optimization (namely of the probability). This is why our approach is different: In a Markov Decision Process, the client would for example specify a feasibility interval for the expected value of the return (= long-term discounted sum of rewards according to some reward function that we explicitly do not assume to be a proper measure of utility), and the learning algorithm would seek a policy that makes the expected return fall anywhere into this interval.

The question of whether an agent somehow necessarily must optimize something is a little philosophical in my view. Of course, given an agent's behavior, one can always find some function that is maximal for the given behavior. This is a mathematical triviality. But this is not the problem we need to address here. The problem we need to address is that the behavior of the agent might get chosen by the agent or its learning algorithm by maximizing some objective function.

It is all about a paradigm shift: In my view, AI systems should be made to achieve reasonable goals that are well-specified w.r.t. one or more proxy metrics, not to maximize whatever metric. What would be the reasonable goal for your modified paperclip maximizer?

Regarding "weakness":

Non-maximizing does not imply weak, let alone "very weak". I'm not suggesting to build a very weak system at all. In fact, maximizing an imperfect proxy metric will tend to give low score on the real utility. Or, to turn this around: The maximum of the actual utility function is most achieved by a policy that does not maximize the proxy metric. We will study this in example environments and report results later this year.

Hi Seth, thank you for your thoughts!

I totally agree that it's just a start, and I hope to have made clear that it is just a start. If it was not sufficiently clear before, I have now added more text making explicit that of course I don't think that dropping the optimization paradigm is sufficient to make AI safe, just that it is necessary. And because if appears necessary and under-explored, I chose to study it for some time.

I don't agree with your 2nd point however: If an agent turns 10% of the world into paperclips, we might still have a chance to survive. If it turns everything into paperclips, we don't.

Regarding the last point:

• Quantilizers are optimizing (namely a certain "regularized" reward function)
• By "surprising amount" you probably mean "surprisingly large amount"? Why is that surprising you then if you agree that they are a "start on taking the points off of the tiger's teeth"? Given the obvious risks of optimization, I am also surprised by the amount of support non-maximization approaches get: namely, I am surprised how small that support is. To me this just shows how strong the grip of the optimization paradigm on people's thinking still is :-)
• I believe any concentration of power is too risky, regardless whether in the hands of a superintelligent AI or dumb human. I have now added some text on this as well.

This is difficult to say. I have a relatively clear intuition what I mean by optimization and what I mean by optimizing behavior. In your example, merely asking for the cheapest flight might be safe as long as you don't automatically then book that flight without spending a moment to think about whether taking that one-propeller machine without any safety belts that you have to pilot yourself is actually a good idea just because it turned out to be the cheapest. I mostly care about agents that have more agency than just printing text to your screen.

I believe what some people call "AI heaven" can be reached with AI agents that don't book the cheapest flights but that book you a flight that costs no more than you specify, take no longer than you specify, and have at least those safety equipment and other facilities that you specify. In other words: satisficing! Another example: Not find me a job that earns me as much income as possible, but find me a job that earns me at least as much income to satisfy all my basic needs and let's me have as much fun from leisure activities as I can squeeze into my lifetime. And so on... 

Regarding "improvement": Replacing a state s by a state s' that scores higher on some metric r, so that r(s') > r(s), is an "improvement w.r.t. r", not an optimization for r. An optimization would require replacing s by that s' for which there is no other s'' with r(s'') > r(s'), or some approximate version of this.

One might think that a sequence of improvements must necessarily constitute an optimization, so that my distinction is unimportant. But this is not correct: While any sequence of improvements r(s1) < r(s2) must make r(sn) converge to some value r° (at least if r is bounded), this limit value r° will in general be considerably lower than the maximal value r* = max r(s). unless the procedure that selects the improvements is especially designed to find that maximum, in other words, is an optimization algorithm. Note that optimization is a hard problem in most real-world cases, much harder than just finding some sequence of improvements.

Dear Will,

thanks for these thoughtful comments. I'm not sure I understand some aspects of what you say correctly, but let me try to make sense of this in the example of Zhuang et al., http://arxiv.org/abs/2102.03896. If the utility function is defined only in terms of a proper subset of the attributes, it will exploit the seemingly irrelevant remaining attributes in the optimization, whether or not some of the attributes it uses represent conflicting goals. Even when conflicting goals are "present across all dimensions of the agent's utility function", that utility function might simple ignore relevant side-effects, e.g. because the designers and teachers have not anticipated them at all.

Their example in Fig. 2 shows this nicely. In contrast, with a satisficing goal of achieving only, say, 6 in Fig. 2, the agent will not exploit the unrepresented features as much and actual utility will be much larger.