@beren discusses the assumption that intelligent systems would be well factored into a world model, objectives/values and a planning system.
He highlights that this factorisation doesn't describe intelligent agents created by ML systems (e.g. model free RL) well. Model free RL agents don't have cleanly factored architectures but tend to learn value functions/policies directly from the reward signal.
Such systems are much less general than their full model based counterpart as policies they learned that are optimal under one reward function may perform very poorly under another reward function.
Yet, contemporary ML favours such systems over their well factored counterparts because the are much more efficient:
- Inference costs can be paid up front by learning a function approximator of the optimal policy and amortised over the agent's lifetime
- A single inference step can be performed as a forward pass through the function approximator in a non factored system vs searching through a solution space to determine the optimal plan/strategy for well factored systems
- The agent doesn't need to learn features of the environment that aren't relevant to their reward function
- The agent can exploit the structure of the underlying problem domain
- Specific recurring patterns can be better amortised
Beren attributes this tradeoff between specificity and generality to no free lunch theorems.
Attaining full generality is prohibitively expensive; as such full orthogonality is not the default or ideal case, but merely one end of a pareto tradeoff curve, with different architectures occupying various positions along it.
The future of AGI systems will be shaped by the slope of the pareto frontier across the range of general capabilities, determining whether we see fully general AGI singletons, multiple general systems, or a large number of highly specialised systems.
I think the way the orthogonality thesis is typically used in arguments might be closer to his definition than to yours.
Your definition is trivially true: all it requires is that an AGI having a specified goal is not physically impossible. But that doesn't prove that all goals are equally likely to occur, or even that AGI will have "goals" at all.
The way I see it deployed in practice is to say that a "dumb" AI will have some silly goal like "build squiggles", will go through an intelligence scale-up, and will keep that goal in hyper-intelligent form. (and then pursuing that goal will result in disaster).
This argument doesn't necessarily work if goals and intelligence at tasks are highly correlated, as they are currently for deep learning systems. It may be that in practical terms, scaling up in intelligence requires at least partially giving up on your initial goals. Or conversely, that only AI's with certain types of goals will ever succeed at scaling themselves up in intelligence.