Every public company in America has a legally-mandated obligation to maximize shareholder returns

This is false.  (The analogy between corporations and unaligned AGI is misleading for many other reasons, of course, not the least of which is that corporations are not actually coherent singleton agents, but are made of people.)

Profit is residual income after inputs, wages and interests are paid. It is a signal that says “this firm is successfully deploying capital to serve the welfare of the consumer” (consumer=person weighted by wealth).

Of course, if the society does not give rights to some sentient beings, and torturing them is an intermediate step in the production of valued goods, it will make profits. In the past, when slave labour was a legal input, capitalists used it, and when the input became unavailable, they simply made profits using other available inputs.

Capitalism is a social mechanism. It allows social coordination in a massive and extremely efficient way. It empowers, multiplies and carry to the extreme existing social values and preferences.