All of Kaj_Sotala's Comments + Replies

Thanks for noticing that! These days I would always use "their" in this kind of context, but I guess I didn't yet have that habit back in 2014. Edited.

(Upvoted.)

Some of the attempts within EA to solve this seem to be to push even more towards just being a professional network. I think that's dangerously wrong, because it doesn't remove the informal networks and their power. It just makes access to them harder, and people more desperate to get in.

Somewhat relevant counterpoint:

For everyone to have the opportunity to be involved in a given group and to participate in its activities the structure must be explicit, not implicit. The rules of decision-making must be open and available to everyone, and this ca

gwern on /r/machinelearning:

There's no comparison to prior full-press Diplomacy agents, but if I'm reading the prior-work cites right, this is because basically none of them work - not only do they not beat humans, they apparently don't even always improve over themselves playing the game as if it was no-press Diplomacy (ie not using dialogue at all). That gives an idea how big a jump this is for full-press Diplomacy.

There's a commentated video by someone who plays as the only human in an otherwise all-Cicero game, which at least makes it seem like the dialogue is doing a lot.

Worth noting that this was "Blitz" Diplomacy with only five-minute negotiation rounds. Still very impressive though.

Some behavioral traits such as general intelligence show very high heritability – over 0.70 – in adults, which is about as heritable as human height.

I'm very confused about what numbers such as this mean in practice, since the most natural interpretation ("70% of the trait is genetically determined") is wrong, but there aren't very many clear explanations of what the correct interpretation is. When I tried asking this on LW, the top-voted answer was that it's a number that's mostly useful if you're doing animal breeding, but probably not useful for much el... (read more)

I'm not sure if there is any reason that should be strongly persuasive to a disinterested third party, at the moment. I think the current evidence is more on the level of "anecdotally, it seems like a lot of rationalists and EAs get something out of things like IFS". 

But given that one can try out a few sessions of a therapy and see whether you seem to be getting anything out of it, that seems to be okay?  Anecdotal evidence isn't enough to strongly show that one should definitely do a particular kind of therapy. But it can be enough to elevate a therapy to the level of things that might be worth giving a shot to see if anything comes out of it.

If they are better, why haven't they been more widely adopted by mainstream medicine?

Part of it is that the effectiveness of therapy is often hard and slow to study, so it's hard to get unambiguous evidence of one being better than another. E.g. many therapists, even if working within a particular school of therapy, will apply it in an idiosyncratic style that draws upon their entire life/job experience and also knowledge of any other therapy styles they might have. That makes it hard to verify the extent to which all the therapists in the study really are... (read more)

Oh wow, that is a really great paper! Thank you very much for linking it.

why anyone would choose one big ritual like 'Doom circles' instead of just purposefully inculcating a culture of opennes to giving / receiving critique that is supportive and can help others?

These don't sound mutually exclusive to me; you can have a formal ritual about something and also practice doing some of the related skills on a more regular basis.

That said, for many people, it would be emotionally challenging if they needed to be ready to receive criticism all the time. A doom circle is something where you can receive feedback at such a time when you... (read more)

That's correct, most of the people in the circle (including the person with the wizard line, I think) I'd only met a couple of days before.

That's great, thank you :D 

Yes please :D

It's my impression that in writing workshops where people bring their writing to be criticized, it's also a common rule that the writers are not allowed to respond to the feedback. I believe the rule exists exactly because of what you say: because another person's feedback may be off or biased for a variety of reasons. If there was a discussion about it, the recipient of the feedback might get defensive and want to explain why the feedback was flawed. That would risk the conversation taking an unpleasant tone and also any correct feedback not being properl... (read more)

I would like to encourage environments in EA  which are mutually supportive and kind.

For what it's worth, my experience of Doom Circles is that they felt explicitly supportive and kind. It felt like other people were willing to pay a social cost to give me honest feedback in a way that they would otherwise feel hesitant to do, and I appreciated them doing that for me.

The incentives are to provide harsh critique to appear novel and insightful of the other person. 

I wouldn't say that there's an incentive to be harsh, since while you are providing f... (read more)

I feel like this post is lacking an explanation of what's good about this practice, so I'll share my experiences.

I think I've attended a couple of Doom Circles (weirdly, my memory claims that there would have been one at my 2014 CFAR workshop but the post says they were first done is 2015, so maybe I'm mixing that up with some later event). I've usually thought of myself as pretty thin-skinned (much less these days but much more back then), but my recollection is that the experience was clearly positive. 

It's very rare to actually get critical feedbac... (read more)

Huh! That's surprising.

Fair. In that case this seems like a necessary prerequisite result for doing that deeper investigation, though, so valuable in that respect.

At least for myself, it wouldn't have been obvious in advance that there would be exactly two factors, as opposed to (say) one, three or four.

Perhaps more educated people are more happy with their career and thus more reluctant to change it?

Or just more invested in it - if you've spent several years acquiring a degree in a topic, you may be quite reluctant to go do something completely different.

For future studies, might be worth rephrasing this item in a way where this doesn't act as a confounder for the results? I'd expect people in their early twenties to answer it quite differently than people in their early forties.

I was thinking that if they insist on requiring it (and I get around actually participating), I'll just iterate on some prompts on wombo.art or similar until I get something decent. 

Because it also mentions woo, so I think it’s talking about a broader class if unjustified beliefs than you think.

My earlier comment mentioned that "there are also lots of different claims that seem (or even are) irrational but are pointing to true facts about the world." That was intended to touch upon "woo"; e.g. meditation used to be, and to some extent still is, considered "woo", but there nonetheless seem to be reasonable grounds to think that there's nonetheless something of value to be found in meditation (despite there also being various crazy clai... (read more)

What makes you think it isn't? To me it seems both like a reasonable interpretation of the quote (private guts are precisely the kinds of positions you can't necessarily justify, and it's talking about having beliefs you can't justify) as well as a dynamic that feels like one that I recognize as one that has been occasionally present in the community. Fortunately posts like the one about private guts have helped push back against it.

Even if this interpretation wasn't actually the author's intent, choosing to steelman the claim in that way turns the essay into a pretty solid one, so we might as well engage with the strongest interpretation of it.

There are a few different ways of interpreting the quote, but there's a concept of public positions and private guts. Public positions are ones that you can justify in public if pressed on, while private guts are illegible intuitions you hold which may nonetheless be correct - e.g. an expert mathematician may have a strong intuition that a particular proof or claim is correct, which they will then eventually translate to a publicly-verifiable proof. 

As far as I can tell, lizards probably don’t have public positions, but they probably do have private g

This is indeed a wonderful story!

This version has nicer line breaks, in my opinion.

Here's an audio version read by Leonard Nimoy.

Draft and re-draft (and re-draft). The writing should go through many iterations. You make drafts, you share them with a few people, you do something else for a week. Maybe nobody has read the draft, but you come back and you’ve rejuvenated your wonderful capacity to look at the work and know why it’s terrible.

Kind of related to this: giving a presentation about the ideas in your article is something that you can use as a form of a draft. If you can't get anyone to listen to a presentation, or don't want to give one quite yet, you can pick some people whos... (read more)

Depends on exactly which definition of s-risks you're using; one of the milder definitions is just "a future in which a lot of suffering exists", such as humanity settling most of the galaxy but each of those worlds having about as much suffering as the Earth has today. Which is arguably not a dystopian outcome or necessarily terrible in terms of how much suffering there is relative to happiness, but still an outcome in which there is an astronomically large absolute amount of suffering.

Fair point. Though apparently measures of 'life satisfaction' and 'meaning' produce different outcomes:

So, how did the World Happiness Report measure happiness? The study asked people in 156 countries to “value their lives today on a 0 to 10 scale, with the worst possible life as a 0 and the best possible life as a 10.” This is a widely used measure of general life satisfaction. And we know that societal factors such as gross domestic product per capita, extensiveness of social services, freedom from oppression, and trust in government and fellow

It has been suggested that people are succumbing to a focusing illusion when they think that having children will make them happy, in that they focus on the good things without giving much thought to the bad.

Worth noting that you might get increased meaningfulness in exchange for the lost happiness, which isn't necessarily an irrational trade to make. E.g. Robin Hanson:

Stats suggest that while parenting doesn’t make people happier, it does give them more meaning. And most thoughtful traditions say to focus more on meaning that happiness. Meaning is how you

Thanks. It looks to me that much of what's being described at these links is about the atmosphere among the students at American universities, which then also starts affecting the professors there. That would explain my confusion, since a large fraction of my academic friends are European, so largely unaffected by these developments.

there could be a number of explanations aside from cancel culture not being that bad in academia.

I do hear them complain about various other things though, and I also have friends privately complaining about cancel culture in non-academic contexts, so I'd generally expect this to come up if it were an issue. But I could still ask, of course.

We also discussed some possible reasons for why there might be a disappointing future in the sense of having a lot of suffering, in sections 4-5 of Superintelligence as a Cause or Cure for Risks of Astronomical Suffering. A few excerpts:

4.1 Are suffering outcomes likely?

Bostrom (2003a) argues that given a technologically mature civilization capable of space colonization on a massive scale, this civilization "would likely also have the ability to establish at least the minimally favorable conditions required for future lives to be worth living",

yet academia is now the top example of cancel culture

I'm a little surprised by this wording? Certainly cancel culture is starting to affect academia as well, but I don't think that e.g. most researchers think about the risk of getting cancelled when figuring out the wording for their papers, unless they are working on some exceptionally controversial topic?

I have lots of friends in academia and follow academic blogs etc., and basically don't hear any of them talking about cancel culture within that context. I did recently see a philosopher... (read more)

On the positive side, a recent attempt to bring cancel culture to EA was very resoundingly rejected, with 111 downvotes and strongly upvoted rebuttals.

I don't know, but I get the impression that SWB questions are susceptible to framing effects in general: for example, Biswas-Diener & Diener (2001) found that when people in Calcutta were asked for their life satisfaction in general, and also for their satisfaction in 12 subdomains (material resources, friendship, morality, intelligence, food, romantic relationship, family, physical appearance, self, income, housing, and social life), they gave on average a slightly negative rating for the global satisfaction, while also giving positive ... (read more)

In particular, Elon Musk claims that BCIs may allow us to integrate with AI such that AI will not need to outcompete us (Young, 2019). It is unclear at present by what exact mechanism a BCI would assist here, how it would help, whether it would actually decrease risk from AI, or if it is a valid claim at all. Such a ‘solution’ to AGI may also be entirely compatible with global totalitarianism, and may not be desirable. The mechanism by which integrating with AI would lessen AI risk is currently undiscussed; and at present, no serious academi

Let's look at some of your references. You say that Scott has endorsed eugenics; let's look up the exact phrasing (emphasis mine):

Even though I like both basic income guarantees and eugenics, I don’t think these are two things that go well together – making the income conditional upon sterilization is a little too close to coercion for my purposes. Still, probably better than what we have right now.

"I don't like this, though it would probably be better than the even worse situation that we have today" isn't exact... (read more)

Also the sleight of hand where the author implies that Scott is a white supremacist, and supports this not by referencing anything that Scott said, but by referencing things that unrelated people hanging out on the SSC subreddit have said and which Scott has never shown any signs of endorsing. If Scott himself had said anything that could be interpreted as an endorsement of white supremacy, surely it would have been mentioned in this post, so its absence is telling.

As Tom Chivers recently noted:

It’s part of the SSC ethos that “if you don

Not to be rude, but what context do you recommend would help for interpreting the statement, "I like both basic income guarantees and eugenics," or describing requiring poor people to be sterilized to receive basic income as "probably better than what we have right now?"

The part from the middle of that excerpt that you left out certainly seems like relevant context: "Even though I like both basic income guarantees and eugenics, I don’t think these are two things that go well together – making the income conditional ... (read more)

Malevolent humans with access to advanced technology—such as whole brain emulation or other forms of transformative AI—could cause serious existential risks and suffering risks.

Possibly relevant: Machiavellians Approve of Mind Upload Technology Directly and Through Utilitarianism (Laakasuo et al. 2020), though it mainly tested whether machiavellians express moral condemnation of mind uploading, rather than their interest directly.

In this preregistered study, we have two novel findings: 1) Utilitarian moral preferences are strongly and psyc

You seem to be working under the assumption that we have either emotional or logical motivations for doing something. I think that this is mistaken: logic is a tool for achieving our motivations, and all of our motivations ultimately ground in emotional reasons. In fact, it has been my experience that focusing too much on trying to find "logical" motivations for our actions may lead to paralysis, since absent an emotional motive, logic doesn't provide any persuasive reason to do one thing over another.

You said that people act altruistically ... (read more)

My perspective here is that many forms of fairness are inconsistent, and fall apart on significant moral introspection as you try to make your moral preferences consistent. I think the skin-color thing is one of them, which is really hard to maintain as something that you shouldn't pay attention to, as you realize that it can't be causally disentangled from other factors that you feel like you definitely should pay attention to (such as the person's physical strength, or their height, or the speed at which they can run).

I think that a sens... (read more)

On the other hand, there are also arguments for why one should work to prevent extinction even if one did have the kind of suffering-focused view that you're arguing for; see e.g. this article. To briefly summarize some of its points:

If humanity doesn't go extinct, then it will eventually colonize space; if we don't colonize space, it may eventually be colonized by an alien species with even more cruelty than us.

Whether alternative civilizations would be more or less compassionate or cooperative than humans, we can only guess. We may however

Do you have a short summary of why he thinks that someone answering the question of "would you have preferred to die right after child birth?" with "No?" is not strong evidence that they should have been born?

I don't know what Benatar's response to this is, but - consider this comment by Eliezer in a discussion of the Repugnant Conclusion:

“Barely worth living” can mean that, if you’re already alive and don’t want to die, your life is almost but not quite horrible enough that you would rather commit suicide than endure. But if

In the past [EAF/FRI] have been rather negative utilitarian, which I have always viewed as an absurd and potentially dangerous doctrine. If you are interested in the subject I recommend Toby Ord’s piece on the subject. However, they have produced research on why it is good to cooperate with other value systems, making me somewhat less worried.

(I work for FRI.) EA/FRI is generally "suffering-focused", which is an umbrella term covering a range of views; NU would be the most extreme form of that, and some of us do lean that way, but many disagree w... (read more)

The following is roughly how I think about it:

If I am in a situation where I need help, then for purely selfish reasons, I would prefer people-who-are-capable-of-helping-me to act in such a way that has the highest probability of helping me. Because I obviously want my probability of getting help, to be as high as possible.

Let's suppose that, as in your original example, I am one of three people who need help, and someone is thinking about whether to act in a way that helps one person, or to act in a way that helps two people. Well, if they act in a way th... (read more)

Hi Daniel,

you argue in section 3.3 of your paper that nanoprobes are likely to be the only viable route to WBE, because of the difficulty in capturing all of the relevant information in a brain if an approach such as destructive scanning is used.

You don't however seem to discuss the alternative path of neuroprosthesis-driven uploading:

we propose to connect to the human brain an exocortex, a prosthetic extension of the biological brain which would integrate with the mind as seamlessly as parts of the biological brain integrate with each other. [...] we ma

Also, one forthcoming paper of mine released as a preprint; and another paper that was originally published informally last year but published in somewhat revised and peer-reviewed form this year:

• Sotala, Kaj (2018). Disjunctive Scenarios of Catastrophic AI Risk. AI Safety and Security (Roman Yampolskiy, ed.), CRC Press. Forthcoming.
• Sotala, Kaj (2017) How Feasible is the Rapid Development of Artificial Superintelligence? Physica Scripta 92 (11), 113001.

Both were done as part of my research for the Foundational Research Institute; maybe include us in yo... (read more)

There seem to be a lot of leads that could help us figure out the high-value interventions, though: i) knowledge about what causes it and what has contributed to changes of it over time ii) research directions that could help further improve our understanding of what causes it / what doesn't cause it iii) various interventions which already seem like they work in a small-scale setting, though it's still unclear how they might be scaled up (e.g. something like Crucial Conversations is basically about increasing trust and safety in one-to-one and small-group... (read more)

Thanks for the reference! That sounds valuable.

I think whether suffering is a 'natural kind' is prior to this analysis: e.g., to precisely/objectively explain the functional role and source of something, it needs to have a precise/crisp/objective existence.

I take this as meaning that you agree that accepting functionalism is orthogonal to the question of whether suffering is "real" or not?

If it is a placeholder, then I think the question becomes, "what would 'something better' look like, and what would count as evidence that something is better?

What something better would look lik... (read more)