Do I understand you correctly here?

Each agent has a computable partial preference ordering $x\le y$ that decides if it prefers $x$ to $y$.

We'd like this partial relation to be complete (i.e., defined for all $x,y$) and transitive (i.e., $x\le y$ and $y\le z$ implies $x\le z$).

Now, if the relation is sufficiently non-trivial, it will be expensive to compute for some $x,y$. So it's better left undefined...?

If so, I can surely relate to that, as I often struggle computing my preferences. Even if they are theoretically complete. But it seems to me the relationship is still defined, but might not be practical to compute.

It's also possible to think of it in this way: You start out with partial preference ordering, and need to calculate one of its transitive closures. But that is computationally difficult, and not unique either.

I'm unsure what these observations add to the discussion, though.

Some comments:

1. Have you considered hiring a designer for this document? It doesn't look good at all and is filled up with bold faces all over the place.
2. Why is it so long? I don't see why it's important for vegans to know that cows are supplemented with vitamin B12.
3. It could have benefited a lot from lists of key takeaways. For instance, do you need to take vitamin D3 supplementation, and how much? Much of the document feels like an info dump to me.

Sure, if your goal is to be a good writer! But, I'm not worried about that. I just want people to understand me.

As far as I can recall, my paragraphs are usually about half as long when I ask ChatGPT to simplify.

That said, I tend to write in an academic style.

I agree that academic language should be avoided in both forums and research papers.

It might be a good idea for forum writers to use a tool like ChatGPT to make their posts more readable before posting them. For example, they can ask ChatGPT to "improve the readability" of their text. This way, writers don't have to change their writing style too much and can avoid feeling uncomfortable while writing. Plus, it saves time by not having to go back and edit clunky sentences. Additionally, by asking ChatGPT to include more slang or colloquial language, the tool can better match the writer's preferred style. (Written with the aid of ChatGPT in exactly the way I proposed. :p)

If I understand you correctly, what you're proposing is essentially a subset of classical decision theory with bounded utility functions. Recall that, under classical decision theory, we choose our action according to $$\underset{a\in A}{max}E\left[u\right(a,X\left)\right],$$ where $X$ is a random state of nature and $A$ an action space.

Suppose there are $N$ (infinitely many works too) moral theories $s_1,s_2,\dots ,s_N$, each with probability $p\left(s_i\right)$ and associated utility $u_i$. Then we can define $$u(a,X)=\sum _{i=1}^{N}p\left(s_i\right)u_i(a,X).$$ This step gives us (moral) uncertainty in our utility function.

Then, as far as I understand you, you want to define some component utility functions as $$\begin{array}{ccc}{u}_i(a,X)& =& \{\begin{array}{cc}1,& \text{if}(a,X)\text{is acceptable under theory}{s}_i,\\ 0,& \text{if}(a,X)\text{is unacceptable under theory}{s}_i.\end{array}\end{array}$$ As then $0\le E{u}_i(a,X)\le 1$ is the probability of an acceptable outcome under $s_i$. And since we're taking the expected value of these bounded component utilities to construct $u$, we're in classical bounded utility function land.

That said, I believe that

1. This post would benefit from a rewrite of the paragraph starting with "Success maximization is a mechanism by which to generalize maxipok". It states " Let $a_i$ be an action $i$ from the set of $m$ actions $A=a_1,a_2,\dots ,a_m$. " Is $i$ and action, $a_i$ and action, or both? I also don't understand what $\pi$ is. Are there states of nature in this framework? You say that $s$ is a moral theory, so it cannot be $s$?
2. You should add concrete examples. If you add one or two it might become easier to understand what you're doing despite the formal definition not being 100% clear.

Thanks for writing this.

1. I wrote about "decay of predictions" here. I would classify the problem as hard.
2. Do you have a feeling for how suitable the projects are for academic projects? Such as bachelor theses or master theses, perhaps? It would be great to show a list of projects to students!

Could you elaborate?

Sorry, but I don't understand what you mean.

Here's the context I'm thinking about. Say you have two options $Y_a$ and $Y_b$. They have different true expected values $E\left(Y_a\right)$ and $E\left(Y_b\right)$. The market estimates their expectations as $\hat{E}\left(Y_a\right)$ and $\hat{E}\left(Y_b\right)$. And you (or the decider) choose the option with highest estimated expectation. (I was unclear about estimation vs. true values in my previous comment.)

Does this have something to do with your remarks here?

Also, there's always a way to implement "the market decides". Instead of asking P(Emissions|treaty), ask P(Emissions|market advises treaty), and make the market advice = the closing prices. This obviously won't be very helpful if no-one is likely to listen to the market, but again the point is to think about markets that people are likely to listen to.

Potential outcomes are very clearly and rigorously defined as collections of separate random variables, there is no "I know it when I see it" involved. In this case you choose between two options, and there is no conditional probability involved unless you actually need it for estimation purposes.

Let's put it a different way. You have the option of flipping two coins, either a blue coin or a red coin. You estimate the expected probability of heads as $P\left(\text{blue}\right)=0.6$ and $P\left(\text{red}\right)=0.5$. You base your choice of which coin to toss on which probability is the largest. There is actually no need to use scary-sounding terms like counterfactuals or potential outcomes at all, you're just choosing between random outcomes.

We could create a separate market on how the decision market resolves, and it will resolve unambiguously.

That sounds like an unnecessarily convoluted solution to a question we do not need to solve!

However we deal with that, I expect the story ends up sounding quite similar to my original comment - the critical step is that the choice does not depend on anything but the closing price.

Yes, I agree. And that's why I believe we shouldn't use conditional probabilities at all, as it makes it confusion possible.