How can we be sure that there aren’t a couple dozen more zeroes in there?

I think that's a great point! I think the behavioral econ/psychology literature should make us cautious too:

• people overestimate small frequencies [^1]
• and even when told the true probability, people overweight small probabilities (this is standard prospect theory) [^2]

[^1] Lichtenstein, Sarah, Paul Slovic, Baruch Fischhoff, Mark Layman, and Barbara Combs. Judged Frequency of Lethal Events.

[^2] Barberis, Nicholas C. “Thirty Years of Prospect Theory in Economics: A Review and Assessment.” Journal of Economic Perspectives 27, no. 1 (2013): 173–96. https://doi.org/10.1257/jep.27.1.173.

 I think this depends on whether farmed or wild animal welfare matters more. I don't have an answer, so let's treat it as 50/50. 

1. If wild animals matter more, what could happen?  On the upside, AGI might enable us to help wild animals.  On the downside, it might lead to humans creating biospheres on other planets, which would increase the suffering of wild animals by many orders of magnitude.
2. If farmed animals matter more, the upside could be that AGI enables us to substitute farmed animals completely (cultivated meat, etc.). The downside could be that people get richer and want to eat more meat, or that AGI changes the production of farmed animals in a way that increases suffering. 

Again, I don't know whether the upside or downside in each scenario is more likely.  Let's say each is 50/50 again.  I think this makes 1) EV negative and 2) EV positive, with the aggregate being slightly EV negative.

Great post!

I keep thinking about labor shares though. Yes, wages might rise in a world with AGI, so far so good. But I still worry about the implications of a decreasing labor share.

1. I think my main concern is a political economy one. A higher capital share might mean that political power is concentrated among few capital owners, which in turn affects the welfare state and so on. This argument is not new, you can find it in this essay and elsewhere. I might try to turn this into a formal model. Do you know if such work exists already?
2. One could also make a reference point argument: My utility is determined by my consumption, but also by my standing relative to others. I might be worse of then if some people's income explodes, but mine rises only somewhat.