Jonas Hallgren
Posts 14
Comments29
Topic Contributions3
Great post; I enjoyed it.
I've got two things to say, the first one being that GPT is a very nice brainstorming tool as it generates many more ideas than you could yourself that you can then prune from, which is nice.
Secondly, I've been doing "peer coaching" with some EA people using reclaim.ai (not sponsored) to automatically book meetings each week where we take turns being the mentor and mentee answering the five following questions:
- What's on your mind?
- When would today's setting be a success?
- Where are you right now?
- How do you get where you want to go?
- What are the actions/first steps to get there?
- Ask for feedback
I really like the framing of meetings with yourself, I'll definitely try that out.
Isn't estimated value calculated by the probability times the utility and as a consequence isn't the higher risk part wrong if one simply looks at it like this? (20% to 10% would be 10x the impact of 2% to 1%)
(I could be missing something here, please correct me in that case)
I didn't mean it in this sense. I think the lesson you drew from it is fair in general, I was just reacting to the things I felt you pulled under the rug, if that makes sense.
Sorry, Pablo, I meant that I got a lot more epistemically humble, I should have thought about how I phrased it more. It was more that I went from the opinion that many worlds is probably true to: "Oh man, there are some weird answers to the Wigner's friend thought experiment and I should not give a major weight to any." So I'm more like maybe 20% on many worlds?
That being said I am overconfident from time to time and it's fair to point that out from me as well. Maybe you were being overconfident in saying that I was overconfident? :D
I will say that I thought the consciousness p zombie distinction was very interesting and a good example of overconfidence as this didn't come across in my previous comment.
Generally, some good points across the board that I agree with. Talking with some physicist friends helped me debunk the many worlds thing Yud has going. Similarly his animal consciousness stuff seems a bit crazy as well. I will also say that I feel that you're coming off way to confident and inflammatory when it comes to the general tone. The AI Safety argument you provided was just dismissal without much explanation. Also, when it comes to the consciousness stuff I honestly just get kind of pissed reading it as I feel you're to some extent hard pandering to dualism.
I totally agree with you that Yudkowsky is way overconfident in the claims that he makes. Ironically enough it also seems that you to some extent are as well in this post since you're overgeneralizing from insufficient data. As a fellow young person, I recommend some more caution when it comes to solid claims about stuff where you have little knowledge (you cherry-picked data on multiple occasions in this post).
Overall you made some good points though, so still a thought-provoking read.
Maybe frame it more as if you're talking to a child. Yes you can tell the child to follow something but how are you certain that it will do it?
Similarly, how can we trust the AI to actually follow the prompt? To trust it we would fundamentally have to understand the AI or safeguard against problems if we don't understand it. The question then becomes how your prompt is represented in machine language, which is very hard to answer.
To reiterate, ask yourself, how do you know that the AI will do what you say?
I'm getting the vibe that your priors are on the world to some extent, being in a multipolar scenario in the future. I'm interested in more specifically what your predictions are for multipolarity versus singleton given the shard-theory thinking as it seems unlikely for recursive self-improvement to happen in the way described given what I understand of your model?