This is very nice! 

I've been thinking that there's a nice generalisable analogy between bayesian updating and forecasting. (It is quite no shit when you think about it but it feels like people aren't exploiting it?)

I'm doing a project on simulating a version of this idea but in a way that utilizes democratic decision making called Predictive Liquid Democracy (PLD) and I would love to hear if you have any thoughts on the general setup. It is model parameterization but within a specific democratic framing.

PLD is basically saying the following:

What if we could set up a trust based meritocratic voting network based on the predictions about how well a candidate will perform? It is futarchy with some twists.

Now for the generalised framing in terms of graphs that I'm thinking of:

1. As an example, if we look at a research network we can say that they're trying to optimise for a certain set of outcomes (citations, new research) and they're trying to make predictions that are going to work. P(U|A)
2. From a system perspective it is hard to influence the nodes even though it is possible. We therfore say that the edges of the graph that is the research network is what we'll optimise. We can then set up a graph that has the signals and graph connections optimised to reach the truth.
3. Since we don't care about the nodes we can also use AIs to help in a combination with human experts. 

I'm writing a paper on setting up the variational mathematics behind this right now. I'm also writing a paper on some more specific simulations of this to run so I'm very grateful for any thoughts you might have of this setup! 

Some people might find that this post is written from a place of agitation which is fully okay. I think that even if you do there are two things that I would want to point out as really good points:

1. A dependence on funders and people with money as something that shapes social capital and incentives, therefore thought in itself. We should therefore be quite vary of the effect that has on people, this can definetely be felt in the community and I think it is a great point.
2. That the karma algorithm could be revisited and that we should think about what incentives are created for the forum through it. 

I think there's a very very interesting project of democratizingthe EA community in a way that makes it more effective. There are lots of institutional design that we can apply to ourselves and I would be very excited to see more work in this direction!

Edit:

Clarification on why I believe it to cause some agitation for some people:

1. I remember that some of the situation around Cremer being a bit politically loaded and that the emotions were running hot at that time and so citing that specific situation makes it lack a bit of context.
   1. There are some object level things that people within the community disagree with when it comes to these comments that point at deeper issues of epistemics and cause prioritization that is actually difficult to answer.
   2. The post makes it seem more one-sided than that situation was. Elitism in EA is something covered in the in-depth fellowship for example and there's a bunch of back and forth there but it is an issue that you will arrive at different consequences on depending on what modelling assumptions you do.
   3. I don't want to make a value judgement on this here, I just want to point out that specifice piece of Cremer's writing has always felt a bit thorny which makes the references feel a bit inflammatory?
   4. For me it's the vibe that it is written from a perspective of being post EA and something about when leaving something behind you want to get back at the thing itself by pointing out how it's wrong? So it is kind of written from a emotionally framed perspective which makes the epistemics fraught?
2. There's some sort of degree where the framing of the post in itself pattern matches onto other critiques that have felt bad faith and so it is "inflammatory" that it raises the immune system of people reading it. I do still think it is quite a valuable point, it is just that part of the phrasing makes it come across more like this than it has to be?
   1. I think that might be because of LLMs often liking to argue towards a specific point but I'm not sure? 
      (You've got some writing that is reminiscent of claude so I could spot the use of it: e.g): 

This isn’t just a technical issue. This is a design philosophy — one that rewards orthodoxy, punishes dissent, and enforces existing hierarchies.

I liked the post, I think it made a good point, I strong upvoted it but I wanted to mention it as a caveat.

I felt that this post might be relevant for longtermism and person affecting views so I had claude write up a quick report on that:

In short: Rejecting the SWWM 💸11% pledge's EV calculation logically commits you to person-affecting views, effectively transforming you from a longtermist into a neartermist.

Example: Bob rejects investing in a $500 ergonomic chair despite the calculation showing 10^50 * 1.2*10^-49 = 12 lives saved due to "uncertainty in the probabilities." Yet Bob still identifies as a longtermist who believes we should value future generations. This is inconsistent, as longtermism fundamentally relies on the same expected value calculations with uncertain probabilities that SWWM uses.

The 🔮 Badge
If you've rejected the SWWM 💸11% Pledge while maintaining longtermist views, we'd appreciate if you could add the 🔮 "crystal ball" emoji to your social media profiles to signal your epistemic inconsistency.

FAQ
Why can't I reject SWWM but stay a longtermist? Both longtermism and SWWM rely on the same decision-theoretic framework of accepting tiny probabilities of affecting vast future populations. Our analysis shows the error bars in SWWM calculations (±0.0000000000000000000000000000000000000000000001%) are actually narrower than the error bars in most longtermist calculations.

What alternatives do I have?

1. Accept the SWWM 💸11% pledge (consistent longtermist)
2. Reject both SWWM and longtermism (consistent person-affecting view)
3. Add the 🔮 emoji to your profile (inconsistent but transparent)

According to our comprehensive Fermi estimate, maintaining consistency between your views on SWWM and longtermism is approximately 4.2x more philosophically respectable.

First and foremost, I'm low confidence here. 

I will focus on x-risk from AI and I will challenge the premise of this being the right way to ask the question.

What is the difference between x-risk and s-risk/increasing the value of futures? When we mention x-risk with regards to AI we think of humans going extinct but I believe that to be a shortform for wise compassionate decision making. (at least in the EA sphere) 

Personally, I think that x-risk and good decision making in terms of moral value might be coupled to each other. We can think of our current governance conditions a bit like correction systems for individual errors. If they pile up, we go off the rail and increase x-risk as well as chances of a bad future. 

So a good decision making system should both account for x-risk and value estimation, therefore the solution is the same and it is a false dichotomy? 

(I might be wrong and I appreciate the slider question anyway!)

First and foremost, I agree with the point. I think looking at this especially from a lens of transformative AI might be interesting. (Coincidentally this is something I'm currently doing using ABMs with LLMs)

You probably know this one but here's a link to a cool project: https://effectiveinstitutionsproject.org/

Dropping some links below, I've been working on this with a couple of people in Sweden for the last 2 years, we're building an open source platform for better democratic decision making using prediction markets:

https://digitaldemocracy.world/flowback-the-future-of-democracy/

The people I'm working with there are also working on:

https://monreform.org/

I know the general space here so if anyone is curious I'm happy to link to people doing different things! 

You might also want to check out: 

https://metagov.org/

FWIW, I find that if you analyze places where we've successfully aligned things in the past (social systems or biology etc.) you find that the 1th and 2nd types of alignment really don't break down in that way. 

After doing Agent Foundations for a while I'm just really against the alignment frame and I'm personally hoping that more research in direction will happen so that we get more evidence that other types of solutions are needed. (e.g alignment of complex systems such as has happened in biology and social systems in the past)

FWIW, I completely agree with what you're saying here and I think that if you seriously go into consciousness research and especially for what we westerners more label as a sense of self rather than anything else it quickly becomes infeasible to hold a position that the way we're taking AI development, e.g towards AI agents will not lead to AIs having self-models. 

For all matters and purposes this encompasses most theories of physicalist or non-dual theories of consciousness which are the only feasible ones unless you want to bite some really sour apples. 

There's a classic "what are we getting wrong" question in EA and I think it's extremely likely that we will look back in 10 years and say, "wow, what are we doing here?". 

I think it's a lot better to think of systemic alignment and look at properties that we want for the general collective intelligences that we're engaging in such as our information networks or our institutional decision making procedures and think of how we can optimise these for resillience and truth-seeking. If certain AIs deserve moral patienthood then that truth will naturally arise from such structures. 

(hot take) Individual AI alignment might honestly be counter-productive towards this view.

So I'll just give some reporting on a vibe I've been feeling on the forum.

I feel a lot more comfortable posting on LessWrong compared to the EA forum because it feels like there's a lot more moral outrage here? Like if I go back 3 or 4 years I felt that the forum was a lot more open to discussing and exploring new ideas. There's been some controversies recently around meat-eater problem stuff and similar and I can't help but just feel uncomfortable posting stuff with how people have started to react? 

I like the different debate weeks as I think it sets up a specific context to create more content which is quite great. Maybe it's a vibe thing, maybe it's something else but I feel that the virtue of open-hearted truthseeking is missing compared to a couple of years back and it makes me want to avoid posting. 

I do believe that the post standard should be lowered at least a bit and for things to be more exploratory again. So uhhhm, more events that invite more community writing and engagement?