We've only known about evolution for two centuries, and the modern prosperous environment (compared to the ancestral environment), post-demographic-transition, is about equally new. This is the first time in evolutionary history that someone who wants to have a dozen children can pretty much just do that.

In the ancestral environment, the fittest humans weren't the ones who wanted to have the most children, or who thought it was moral to maximize their genetic influence over the future. They were the ones who were good at making tools, or good at making friends with people who make the tools, etc. That's different now. Over millions of generations of simulated evolution, I expect the gene pool to become overwhelmingly dominated by people who want to have a lot of kids.

I, however, also think that the creator believing X is true is some evidence that X is true, since the creator would reasonably have more power and knowledge, which makes me guess that they might be more likely than us to be right (hence the update toward their view).

This is only evidence for moral realism inasmuch as the creator has access to evidence about the true morality. If the creator went to you and said "here is why I know Kantianism is true" and then gave some evidence, then that evidence would be evidence.

Basically, if the creator has good justification for their belief, they should be able to give you that justification. If they can't, then why does their belief mean anything?

I think this assumes that moral intuitions are entirely downstream of the design of the mind.

It is not clear to me that there's a way to set up this experiment that doesn't bake in the answer from the beginning.

1. If you design minds as idealized versions of already-existing minds (a la CEV), then they'll be starting from the same moral intuitions. At least among extant human minds, moral intuitions don't vary by all that much.
2. If you run natural selection on highly intelligent minds, they will inevitably converge on the belief that the true morality is to maximize how many descendants they have—because minds that hold that belief will, in the long run, out-compete minds that don't. It does not follow that the true morality is to maximize how many descendants you have.

If you have AI super-philosophers reasoning about ethics, then I'd think the relevant evidence wouldn't be the fact that you can get different AIs to agree with each other, but the actual content of their reasoning. Like, the evidence for moral realism would be that a super-philosopher comes up with an argument about what morality is, and that argument itself contains evidence.

Imagine we point a telescope at some far-off galaxy and find "KANT IS RIGHT ABOUT EVERYTHING" written in stars, in clean sans-serif and at a resolution of 1,000 stars per degree. That would make Kantian ethics seem more likely to be true.

Disagree. It would be evidence that the creator of the universe believes Kantian ethics is true. It would only be evidence if we have reason to believe that the creator of the universe is especially likely to be right about ethics, which I don't think we do.

AI super-philosophers, built a few different ways, agreeing on the same ethics.

Compared to the idea of different human philosophers taking different approaches and ultimately agreeing on ethics, it seems to me that this is only different in degree, in that the super-philosophers are better at philosophy.

Minds with different architectures converging on the same moral intuitions.

This seems highly contingent on what space of minds you look at. If you have the ability to design minds, then it should be trivial to cause those minds to believe any ethical framework you want them to believe.

They probably just came from LessWrong and got the ordering of the buttons mixed up

On LessWrong, the disagree button is on the left and the agree button is on the right. On EAF, the agree button is on the left and the disagree button is on the right.

From this post, I am not clear on what your idea is or what you intend to accomplish. Why does the world need another AI company, and what exactly would this company work on?

Realistically, if you don't have significant experience in the machine learning field (or at least in business), you are going to have a hard time starting a new AI company.

As I understand the term, that sort of ASI wouldn't be considered "misaligned", it would be "aligned, but to the wrong target". I think of misalignment as when you wanted the ASI to do one thing, but it did something else instead.

Two quick thoughts:

1. This is a neat idea, it's difficult to come up with safe preferences to encode in an ASI, and the concept of strong risk-aversion might help.
2. A major obstacle (which I didn't see listed in section 8) is that currently we have no idea how to embed any set of preferences whatsoever in an ASI. 2b. If we figure out how to encode risk-averse preferences in an ASI, then I'm not sure it makes sense to speak of it as "misaligned", because clearly we do know how to get it to pursue goals that we care about. It seems weird to expect that we won't know how to make ASI not want to tile the universe with paperclips, but we will know how to make it want to risk-aversely tile the universe with paperclips.

I think section 10 is pointing at something similar. I find it at least somewhat plausible that RL on risk aversion generalizes better than other kinds of RL. I would still be surprised if we could get risk aversion to generalize to ASI using anything resembling current techniques, but this seems like a better-than-average idea for preventing AI takeover.