Denis Drescher

What would Mary Sue do?

Denis Drescher's Comments

Denis Drescher's Shortform

Studies on Slack” by Scott Alexander: Personal takeaways

There have been studies on how software teams use Slack. Scott Alexander’s article “Studies on Slack” is not about that. Rather it describes the world as a garlic-like nesting of abstraction layers on which there are different degrees of competition vs. cooperation between actors; how they emerged (in some cases); and what their benefit is.

The idea, put simply, at least in my mind, is that in a fierce competition innovations need to prove beneficial immediately in logical time or the innovator will be outcompeted. But limiting innovations to only those that either consist of only one step or whose every step is individually beneficial is, well, limiting. The result are innovators stuck in local optima unable to reach more global optima.

Enter slack. Somehow you create a higher-order mechanism that alleviates the competition a bit. The result is that now innovators have the slack to try a lot of multi-step innovations despite any neutral or detrimental intermediate steps. The mechanisms are different ones in different areas. Scott describes mechanisms from human biology, society, ecology, business management, fictional history, etc. Hence the garlic-like nesting: It seems to me that these systems are nested within each other, and while Scott only ever describes two levels at a time, it’s clear enough that higher levels such as business management depend on lower levels such as those that enable human bodies to function.

This essay made a lot of things clearer to me that I had half intuited but never quite understood. In particular it made me update downward a bit on how much I expect AGI to outperform humans. One of mine reasons for thinking that human intelligence is vastly inferior to a theoretical optimum is that I thought evolution could almost only ever improve one step at a time – that it would take an extremely long time for a multi-step mutation with detrimental intermediate steps to happen through sheer luck. Since slack seems to be built into biological evolution to some extent, maybe it is not as inferior as I thought to “intelligent design” like we’re attempting it now.

It would also be interesting to think about how slack affects zero-sum board games – simulations of fierce competition. In the only board game I know, Othello, you can thwart any plans the opponent might have with your next move in, like, 90+% of cases. Hence, I made a (small but noticeable) leap forward in my performance when I switched from analyzing my position through the lens of “What is a nice move I can play?” to “What is a nice move my opponent could now play if it were their turn and how can I prevent it?” A lot of perfect moves, especially early in the game, switch from looking surprising and grotesk to looking good once I viewed them through that lens. So it seems that in Othello there is rarely any Slack. (I’m not saying that you don’t plan multi-step strategies in Othello, but it’s rare that you can plan them such that actually get to carry them out. Robust strategies play a much greater role in my experience. Then again this may be different at higher levels of gameplay than mine.)

Perhaps that’s related to why I’ve seen not particularly smart people yet turning out to be shockingly efficient social manipulators, and why these people are usually found in low-slack fields. If your situation is so competitive that your opponent can never plan more than one step ahead anyway, you only need to do the equivalent of thinking “What is a nice move my opponent could now play if it were their turn and how can I prevent it?” to beat, like, 80% of them. No need for baroque and brittle stratagems like in Skyfall.

I wonder if Go is different? The board is so big that I’d expect there to be room to do whatever for a few moves from time to time? Very vague surface-level heuristic idea! I have no idea of Go strategy.

I’m a bit surprised that Scott didn’t draw parallels to his interest in cost disease, though. Not that I see any clear once, but there got to be some that are worth at least checking and debunking – innovation slowing so that you need more slack to innovate at the same rate, or increasing wealth creating more slack thereby decreasing competition that would’ve otherwise kept prices down, etc.

The article was very elucidating, but I’m not quite able to now look at a system and tell whether it needs more or less slack or how to establish a mechanism that could produce that slack. That would be important since I have a number of EA friends who could use some more slack to figure out psychological issues or skill up on some areas. The EA funds try to help a bit here, but I feel like we need more of that.

Denis Drescher's Shortform

Effective Altruism and Free Riding” by Scott Behmer: Personal takeaways

Coordination is an oft-discussed topic within EA, and people generally try hard to behave cooperatively toward other EA researchers, entrepreneurs, and donors present and future. But “Effective Altruism and Free Riding” makes the case that standard EA advice favors defection over cooperation in prisoner’s dilemmas (and stag hunts) with non-EAs. It poses the question whether this is good or bad, and what can be done about it.

I’ve had a few thoughts while reading the article but found that most of them were already covered in the most upvoted comment thread. I’ll still outline them in the following as a reference for myself, to add some references that weren’t mentioned, and to frame them a bit differently.

The project of maximizing gains from moral trade is one that I find very interesting and promising, and want to investigate further to better understand its relative importance and strategic implications.

Still, Scott’s perspective was a somewhat new one for me. He points out that in particular the neglectedness criterion encourages freeriding: Climate change is a terrible risk but we tend to be convinced by neglectedness considerations that additional work on it is not maximally pressing. In effect, we’re freeriding on the efforts of activists working on climate change mitigation.

What was new to me about that is that I’ve conceived of neglectedness as a cheap coordination heuristic. Cheap in that it doesn’t require a lot of communication with other cooperators; coordination in the sense that everyone is working towards a bunch of similar goals but need to distribute work among themselves optimally; and heuristic in that it falls short insofar as values are not perfectly aligned, momentum in capacity building is hard to anticipate, and the tradeoffs with tractability and importance are usually highly imprecise.

So in essence, my simplification was to conceive of the world as filled with agents like me in values that use neglectedness to coordinate their cooperative work, and Scott conceives of the world as filled with agents very much unlike me in values that use neglectedness to freeride off of each other’s work.

Obviously, neither is exactly true, but I don’t see an easy way to home in on which model is better: (1) I suppose most people are not centrally motivated by consequentialism in their work, and it may be impossible for us to benefit the motivations that are central to them. But then again there are probably consequentialist aspects to most people’s motivations. (2) Insofar as there are aspects to people’s motivations for their work that we can benefit, how would these people wish for their preferences to be idealized (if that is even the framing that they’d prefer to think about their behavior)? Caspar Oesterheld discusses the ins and outs of different forms of idealization in the eponymous section 3.3.1 of “Multiverse-wide Cooperation via Correlated Decision Making.” The upshot is, very roughly, that idealization through additional information seems less doubious than idealization through moral arguments (Scott’s article mentions advocacy for example). So would exposing non-EAs to information about the importance of EA causes lead them to agree that people should focus on them even at the expense of the cause that they chose? (3) What consequentialist preferences should be even take into account – only altruistic ones or also personal ones, since personal ones may be particularly strong? A lot of people have personal preferences not to die or suffer and for their children not to die or suffer, which may be (imperfectly) aligned with catastrophe prevention.

But the framing of the article and the comments was also different from the way I conceive of the world in that it framed the issue as a game between altruistic agents with different goals. I’ve so far seen all sorts of nonagents as being part of the game by dint of being moral patients. If instead we have a game between altruists who are stewards of the interests of other nonagent moral patients, it becomes clearer why everyone is part of the game, their power, but there are a few other aspects that elude me. Is there a risk of double-counting the interests of the nonagent moral patients if they have many altruist stewards – and does that make a difference if everyone does it? And should a bargaining solution only take the stewards’ power into account (perhaps the natural default, for better or worse) or also the number of moral patients they stand up for? The first falls short of my moral intuitions in the case. It may also cause Ben Todd and many others to leave the coalition because the gains from trade are not worth the sacrifice for them. Maybe we can do better. But the second option seems gameable (by pretending to see moral patienthood where one in fact does not see it) and may cause powerful cooperators to leave the coalition if they have a particularly narrow concept of moral patienthood. (Whatever the result, it seems like that this the portfolio that commenters mentioned, probably akin to the compromise utility function that you maximize in evidential cooperation – see Caspar Oesterheld’s paper.)

Personally, I can learn a lot more about these questions by just reading up on more game theory research. More specifically, it’s probably smart to investigate what the gains from trade are that we could realize in the best case to see if all of this is even worth the coordination overhead.

But there are probably also a few ways forward for the community. Causal (as opposed to acausal) cooperation requires some trust, so maybe the signal that there is a community of altruists that cooperate particularly well internally can be good if paired with the option of others to join that community by proving themselves to be sufficiently trustworthy. (That community may be wider than EA and called differently.) That would probably take the shape of newcomers making the case for new cause areas not necessarily based on their appeal to utilitarian values but based on their appeal to the values of the newcomer – alongside an argument that those values wouldn’t just turn into some form of utilitarianism upon idealization. That way, more value systems could gradually join this coalition, and we’d promote cooperation the way Scott recommends in the article. It’ll probably make sense to have different nested spheres of trust, though, with EA orgs at the center, the wider community around that, new aligned cooperators further outside, occasional mainstream cooperators further outside yet, etc. That way, the more high-trust spheres remain even if sphere’s further on the outside fail.

Finally, a lot of these things are easier in the acausal case that evidential cooperation in large worlds (ECL) is based on (once again, see Caspar Oesterheld’s paper). Perhaps ECL will turn out to make sufficiently strong recommendations that we’ll want to cooperate causally anyway despite any risk of causal defection against us. This stikes me as somewhat unlikely (e.g., many environmentalists may find ECL weird, so there may never be many evidential cooperators among them), but I still feel sufficiently confused about the implications of ECL that I find it at least worth mentioning.

What are the leading critiques of "longtermism" and related concepts

“The Epistemic Challenge to Longtermism” by Christian Tarsney is perhaps my favorite paper on the topic.

Longtermism holds that what we ought to do is mainly determined by effects on the far future. A natural objection is that these effects may be nearly impossible to predict—perhaps so close to impossible that, despite the astronomical importance of the far future, the expected value of our present options is mainly determined by short-term considerations. This paper aims to precisify and evaluate (a version of) this epistemic objection. To that end, I develop two simple models for comparing “longtermist” and “short-termist” interventions, incorporating the idea that, as we look further into the future, the effects of any present intervention become progressively harder to predict. These models yield mixed conclusions: If we simply aim to maximize expected value, and don’t mind premising our choices on minuscule probabilities of astronomical payoffs, the case for longtermism looks robust. But on some prima facie plausible empirical worldviews, the expectational superiority of longtermist interventions depends heavily on these “Pascalian” probabilities. So the case for longtermism may depend either on plausible but non-obvious empirical claims or on a tolerance for Pascalian fanaticism.

“How the Simulation Argument Dampens Future Fanaticism” by Brian Tomasik has also influenced my thinking but has a more narrow focus.

Some effective altruists assume that most of the expected impact of our actions comes from how we influence the very long-term future of Earth-originating intelligence over the coming ~billions of years. According to this view, helping humans and animals in the short term matters, but it mainly only matters via effects on far-future outcomes.

There are a number of heuristic reasons to be skeptical of the view that the far future astronomically dominates the short term. This piece zooms in on what I see as perhaps the strongest concrete (rather than heuristic) argument why short-term impacts may matter a lot more than is naively assumed. In particular, there's a non-trivial chance that most of the copies of ourselves are instantiated in relatively short-lived simulations run by superintelligent civilizations, and if so, when we act to help others in the short run, our good deeds are duplicated many times over. Notably, this reasoning dramatically upshifts the relative importance of short-term helping even if there's only a small chance that Nick Bostrom's basic simulation argument is correct.

My thesis doesn't prove that short-term helping is more important than targeting the far future, and indeed, a plausible rough calculation suggests that targeting the far future is still several orders of magnitude more important. But my argument does leave open uncertainty regarding the short-term-vs.-far-future question and highlights the value of further research on this matter.

Finally, you can also conceive of yourself as one instantiation of a decision algorithm that probably has close analogs at different points throughout time, which makes Caspar Oesterheld’s work relevant to the topic. There are a few summaries linked from that page. I think it’s an extremely important contribution but a bit tangential to your question.

[Stats4EA] Expectations are not Outcomes

I’ve found Christian Tarsney’s “Exceeding Expectations” insightful when it comes to recognizing and maybe coping with the limits of expected value.

The principle that rational agents should maximize expected utility or choiceworthiness is intuitively plausible in many ordinary cases of decision-making under uncertainty. But it is less plausible in cases of extreme, low-probability risk (like Pascal's Mugging), and intolerably paradoxical in cases like the St. Petersburg and Pasadena games. In this paper I show that, under certain conditions, stochastic dominance reasoning can capture most of the plausible implications of expectational reasoning while avoiding most of its pitfalls. Specifically, given sufficient background uncertainty about the choiceworthiness of one's options, many expectation-maximizing gambles that do not stochastically dominate their alternatives "in a vacuum" become stochastically dominant in virtue of that background uncertainty. But, even under these conditions, stochastic dominance will not require agents to accept options whose expectational superiority depends on sufficiently small probabilities of extreme payoffs. The sort of background uncertainty on which these results depend looks unavoidable for any agent who measures the choiceworthiness of her options in part by the total amount of value in the resulting world. At least for such agents, then, stochastic dominance offers a plausible general principle of choice under uncertainty that can explain more of the apparent rational constraints on such choices than has previously been recognized.

See also the post/sequence by Daniel Kokotajlo, “Tiny Probabilities of Vast Utilities”. I’m linking to the post that was most valuable to me, but by default it might make sense to start with the first one in the sequence. ^^

Modelers and Indexers

Yeah, totally agree! The Birds and Frogs distinction sounds very similar! I’ve pocketed the original article for later reading.

And I also feel that the Adaptors–Innovators one is “may be slightly correlated but is a different thing.” :-)

Modelers and Indexers

Yes! I’ve been thinking about you a lot while I was writing that post because you yourself strike me as a potential counterexample to the usefulness of the distinction. I’ve seen you do exactly what you describe and generally display comfort in situations that indexers would normally be comfortable in, while at the same time you evidently have quite similar priorities to me. So either you break the model or you’re just really good at both! :-)

Modelers and Indexers

Thank you!

Yeah, that feels fitting to me too. I found these two posts on the term:

https://www.lesswrong.com/posts/xqAnKW46FqzPLnGmH/causal-reality-vs-social-reality https://www.lesswrong.com/posts/j2mcSRxhjRyhyLJEs/what-is-social-reality

A lot of social things appear arbitrary when deep down they must be deterministic. But bridging that gap is perhaps both computationally infeasible and doesn’t lend itself to particularly powerful abstractions (except for intentionality). At the same time, though, the subject is more inextricably integrated with the environment, so that it makes more sense to model the environment as falling into intentional units (agents) who are reactive. And then maybe certain bargaining procedures emerged (because they were adaptive) that are now integrated into our psyche as customs and moral intuitions.

For these bargaining procedures, I imagine, it’ll be important to abstract usefully from specific situations to more general games. Then you can classify a new situation as one that either requires going through the bargaining procedure again or is a near-replication of a situation whose bargaining outcome you already have stored. That would require exactly the indexer types of abilities – abstracting from situations to archetypes and storing the archetypes.

(E.g., if you sell books, there’s a stored bargaining solution for that where you declare a price, and if it’s right, hand over the book and get the money for it, and otherwise keep the book and don’t get the money. But if you were the first to create a search engine that indexes the full-text of books, there were no stored bargaining solutions for that and you had to go through the bargaining procedures.)

It also seems to me that there are people who, when in doubt, tend more toward running through the bargaining procedure, while others instead tend more toward observing and learning established bargaining solutions very well and maybe widening their references classes for games. I associate the first a bit with entrepreneurs, low agreeableness, Australia, and the Pavlov strategy, and the second with me, agreeable friends of mine, Germany/Switzerland, and tit for tat.

Bored at home? Contribute to the EA Wiki!

I love the idea of wikis for EA knowledge, but is there an attempt underway yet to consolidate all the existing wikis, beyond the Wikia one? Maybe you can coordinate some data import with the other people who are running EA wikis.

When the Priority Wiki was launched, (I but much more so) John Maxwell compiled some of the existing wikis here.

I think for one of these wikis to take off, it’ll probably need to become the clear Schelling point for wiki activity – maybe an integration with the concepts platform or the forum and a consolidation of all the other wikis as a basis.

I imagine there’d also need to be a way for active wiki authors to gain reputation points, e.g., in this forum, so wiki editing can have added benefits for CV building. Less Wrong also has forum and wiki, and the forum is a very similar software, so maybe they already have plans for such a system.

Does Utilitarian Longtermism Imply Directed Panspermia?

Oh yeah, I was also talking about it only from utilitarian perspectives. (Except for one aside, “Others again refuse it on deontological or lexical grounds that I also empathize with.”) Just utilitarianism doesn’t make a prescription as to the exchange rate of intensity/energy expenditure/… of individual positive experiences to individual negative experiences.

It seems that reasonable people think the outcome of B might actually be worse than A, based on your response.

Yes, I hope they do. :-)

Sorry for responding so briefly! I’m falling behind on some reading.

Does Utilitarian Longtermism Imply Directed Panspermia?

I think I’m not well placed to answer that at this point and would rather defer that to someone who has thought about this more than I have from the vantage points of many ethical theories rather than just from my (or their) own. (I try, but this issue has never been a priority for me.) Then again this is a good exercise for me in moral perspective-taking or what it’s called. ^^

It seems C > B > A, with the difference between A and B greater than the difference between B and C.

In the previous reply I tried to give broadly applicable reasons to be careful about it, but those were mostly just from Precipice. The reason is that if I ask myself, e.g., how long I would be willing to endure extreme torture to gain ten years of ultimate bliss (apparently a popular thought experiment), I might be ready to invest a few seconds if any, for a tradeoff ratio of 1e7 or 1e8 to 1. So from my vantage point, the r-strategist style “procreation” is very disvaluable. It seems like it may well be disvaluable in expectation, but either way, it seems like an enormous cost to bear for a highly uncertain payoff. I’m much more comfortable with careful, K-strategist “procreation” on a species level. (Magnus Vinding has a great book coming out soon that covers this problem in detail.)

But assuming the agnostic position again, for practice, I suppose A and C are clear cut: C is overwhelmingly good (assuming the Long Reflection works out well and we successfully maximize what we really terminally care about, but I suppose that’s your assumption) and A is sort of clear because we know roughly (though not very viscerally) how much disvalue our ancestors have paid forward over the past millions of years so that we can hopefully eventually create a utopia.

But B is wide open. It may go much more negative than A even considering all our past generations – suffering risks, dystopian-totalitarian lock-ins, permanent prehistoric lock-ins, etc. The less certain it is, the more of this disvalue we’d have to pay forward to get one utopia out of it. And it may also go positive of course, almost like C, just with lower probability and a delay.

People have probably thought about how to spread self-replicating probes to other planets so that they produce everything a species will need at the destination to rebuild a flourishing civilization. Maybe there’ll be some DNA but also computers with all sorts of knowledge, and child-rearing robots, etc. ^^ But a civilization needs so many interlocking parts to function well – all sorts of government-like institutions, trust, trade, resources, … – that it seems to me like the vast majority of these civilizations either won’t get off the ground in the first place and remain locked in a probably disvaluable Stone Age type of state, or will permanently fall short of the utopia we’re hoping for eventually.

I suppose a way forward may to consider the greatest uncertainties about the project – probabilities and magnitudes at the places where things can go most badly net negative or most awesomely net positive.

Maybe one could look into Great Filters (they may exist less necessarily than I had previously thought), because if we are now past the (or a) Great Filter, and the Great Filter is something about civilization rather than something about evolution, we should probably assign a very low probability to a civilization like ours emerging under very different conditions through the probably very narrow panspermia bottleneck. I suppose this could be tested on some remote islands? (Ethics committees may object to that, but then these objections also and even more apply to untested panspermia, so they should be taken very seriously. Then again they may not have read Bostrom or Ord. Or Pearce, Gloor, Tomasik, or Vinding for that matter.)

Oh, here’s an idea: The Drake Equation has the parameter f_i for the probability that existing life develops (probably roughly human-level?) intelligence, f_c that intelligent life becomes detectable, and L for the longevity of the civilization. The probability that intelligent life creates a civilization with similar values and potential is probably a bit less than f_c (these civilizations could have any moral values) but more than the product of the two fs. The paper above has a table that says “f_i: log-uniform from 0.001 to 1” and “f_c: log-uniform from 0.01 to 1.” So I suppose we have some 2–5 orders of magnitude uncertainty from this source.

The longevity of a civilization is “L: log-uniform from 100 to 10,000,000,000” in the paper. An advanced civilization that exists for 10–100k years may be likely to have passed the Precipice… Not sure at all about this because of the risk of lock-ins. And I’d have to put this distribution into Guesstimate to get a range of probabilities out of this. But it seems like a major source of uncertainty too.

The ethical tradeoff question above feels almost okay to me with a 1e8 to 1 tradeoff but others are okay with a 1e3 or 1e4 to 1 tradeoff. Others again refuse it on deontological or lexical grounds that I also empathize with. It feels like there are easily five orders of magnitude uncertainty here, so maybe this is the bigger question. (I’m thinking more in terms of an optimal compromise utility function than in moral realist terms, but I suppose that doesn’t change much in this case.)

In the best case within B, there’s also the question whether it’ll be a delay compared to C of thousands or of tens of thousands of years, and how much that would shrink the cosmic endowment.

I don’t trust myself to be properly morally impartial about this after such a cursory investigation, but that said, I would suppose that most moral systems would put a great burden of proof on the intervention because it can be so extremely good and so extremely bad. But tackling these three to four sources of uncertainty and maybe others can perhaps shed more light on how desirable it really is.

I empathize with the notion that some things can’t wait until the Long Reflection, at least as part in a greater portfolio, because it seems to me that suffering risks (s-risks) are a great risk (in expectation) even or especially now in the span until the Long Reflection. They can perhaps be addressed through different and more tractable avenues than other longterm risks and by researchers with different comparative advantages.

A neglected case above is where weapon X destroys life on earth, earth engages in directed panspermia, but there was already life in the universe unbeknownst to earth. I think we agree that B is superior to this case, and therefore the difference between B and A is greater. The question is does the difference between this case and C surpass that between A and B. Call it D. Is D so much worse than C that a preferred loss is from B to A? I don’t think so.

Hmm, I don’t quite follow… Does the above change the relative order of preference for you, and if so, to which order?

So I guess the implied position would be that we should prepare a biotic hedge in case things get especially dire, and invest more in SETI type searches. If we know that life exists elsewhere in the universe, we do not need to deploy the biotic hedge?

There are all these risks from drawing the attention of hostile civilizations. I haven’t thought about what the risk and benefits are there. It feels like that came up in Precipice too, but I could be mixing something up.

Load More