I did not use an LLM in writing this post.

Plants, generative AI models and thermostats have preferences or goals, but they (probably) do not have conscious experiences. Their preferences are (probably) unconscious. But in my moral theory (called mild welfarism), only conscious preferences matter. A conscious preference is a preference (value or goal) that can be consciously experienced by a conscious (sentient) being. Although just a moment ago you may not have been consciously aware of your preference to stay alive, you could have been made aware of that preference, for example when you were in danger and you felt fear, or when you now reflect on that preference by reading this sentence. Hence, your preference to stay alive is now and was a moment ago a conscious preference.

The reason why preferences matter is straightforward: if you prefer a moral theory according to which preferences do not matter, your preference for that moral theory cannot matter either. But why should we only take into consideration conscious preferences (and not unconscious preferences) in our moral deliberations?

The theory of mild welfarism starts with the premise that only welfare matters. By definition, the welfare of a conscious, sentient being in a situation or outcome measures that individual’s strength of preference for that situation or outcome. In other words, welfare measures how good that situation is according to the individual itself. Hence, welfare is tightly linked to preferences. But an individual can only measure the strength of its preferences when it is consciously aware of its preferences.

The ability to pay attention is crucial: a preference only matters when there is someone (a sentient being) who can pay attention to that preference. And the ability to pay attention is the basis of consciousness, according to Michael Graziano’s Attention Schema Theory (AST) of consciousness. According to that theory, consciousness is the brain's simplified, schematic description of its own data-processing focus. To manage its own limited processing power, the brain uses attention to focus on specific signals. Signals can be perceptions or memories, but also goals or preferences. The brain builds an internal map or schema of that attention process in order to control its own internal focus. This attention schema is what we experience as awareness. We feel like we have conscious, subjective awareness because our brains are constantly building a model of our own attention.

The strength of a preference measures how strongly the entity tends to pay attention to the preference. A preference is a signal (in particular information about a goal function), and a strong preference is a signal that is (or tends to be) strongly amplified in order to gain or attract attention when the preference is at stake. Your preference to stay alive is strong, because when your live is at stake in a dangerous situation, the perception of the danger, the fear and the preference to stay alive all grab your attention. The information that staying alive is one of your goals, gets amplified. Without an attention mechanism, all preferences of the unconscious entity have zero strength, as with complete indifference. In the moral calculation, they count as zero. 

In summary: a preference is only relevant in our moral considerations when the entity that has that preference is able to care about that preference, by being able to pay attention to that preference. In order to pay attention, such an entity needs to have an attention control mechanism or attention schema. Such a mechanism automatically and unavoidably generates a consciousness. That is why only conscious preferences matter. For unconscious entities, their preferences have zero strength and hence zero moral relevance.