Seems to me that the underlying reasons, beyond the externalities Buck mentioned, are if your personal ethics have some combination of the following four qualities:
I am like this. I'm pretty sure oge is like this. It seems like there may be some philosophical barriers here. I definitely think that hedonistic utilitarianism violates 1 and 2 outright, and strongly discourages 3 and 4. As a preference utilitarian, the strength of 1 and 4 outweigh the costs to me.