I had a look, it seems to presume the AI-owners will control all the resources, but this doesn't seem like a given (though it may pan out that way). 

I realise you said you didn't want to debate these assumptions, but just wanted to point out that the picture painted doesn't seem inevitable.

I don't really follow why one set of entities getting AGI and not sharing it should necessarily lead to widespread destitution.

Suppose A, B and C are currently working and trading between each other. A develops AGI and leaves B and C to themselves. Would B and C now just starve? Why would that necessarily happen? If they are still able to work as before, they can do that and trade with each other. They would become a bit poorer due to needing to replace the goods that A had a comparative advantage in producing I guess.

For B and C to be made destitute directly, it would seem to require that they are prevented at working at anything like their previous productivity eg if A were providing something essential and irreplaceable for B and C (maybe software products if A is techy?) or if A's AGI went and pushed B and C off a large fraction of natural resources. It doesn't seem very likely to me that B and C couldn't mostly replace what A provided (eg with current open-source software). For A to push B and C off a large enough amount of resources, when the AGI has presumably already made A very rich, would require A to be more selfish and cruel than I hope is likely - but it's unfortunately not unthinkable.

Of course there would probably still be hugely more inequality - but that doesn't imply B and C are destitute.

I could imagine there being indirect large harms on B and C if their drop in productivity were large enough to create a depression, with financial system feedbacks amplifying the effects.

In any case, the picture you paint seems to require an additional reason that B and C cannot produce the things they need for themselves.

Are there roles in your current organisation that you think would be more enjoyable and could move into, say more at the level of making direct contributions?

Also, have you very thoroughly thought through the risks of retiring on $700k? I've seen in various discussions that it's common for people to think that a 4% withdrawal rate is likely sustainable to enable early retirement with low risk, but there are various reasons why that's probably optimistic, so just thought I'd flag it in case that's what this is based on. Maybe it's not...

My understanding of these "reasoning" approaches is that they seem to work very well on problems where there is a well-defined correct answer, and where that can be automatically verified. And it seems reasonable to expect much progress in that area.

What is the thinking of how much of human reasoning work is to do with problems like these?

As a counter-example, in my own particular work on climate prediction, we do not get rapid feedback about what works well, and it is contested what methods and frameworks we should even use i.e. it's not possible presently to say "getting a good answer just requires solving [list of well-defined problems]" (except making computers so fast that we can do pretty much exact simulations of physics). So it doesn't seem clear to me that these reasoning models will get a lot better at that kind of thing. But this is perhaps towards the far end of the spectrum of complex problems.

I can see these reasoning models becoming very good at things like writing code where requirements to be met can be precisely specified and automatically verified, and improving performance of devices (such as computer chips) according to well-specified benchmarks. How much difference would it make to make fast progress on problems similar to these?

There doesn't look to me to be a reason to think that systems trained this way will yield impressive performance at solving messier problems without clear right answers, like predicting complex systems (that can't be observed experimentally or simulated very well), selecting amongst decision options with different strengths on multiple criteria, dealing with organisational politics etc. Does that seem fair?

These are genuine questions - I don't feel I have a good grasp of what kinds of work most of our economy is engaged in...

It's not clear to me why the aim ought to be to sample randomly amongst all people - it seems like a different population could reasonably be chosen!

Sounds interesting. I had a go at the tool, but was a bit perplexed that the "lottery story" it showed me was for a Romanian earning $2,500/month, which doesn't seem like the kind of life that people's attention needs to be most drawn to or represents people that would be helped by effective development charities (it even says this person is at the 86th percentile of global income). And then below that it talked about ending hunger, eradicating disease etc., which didn't relate to the story. I'd focus it on stories about the kinds of people that effective charities would actually help. I tried to get it to generate another story to see what else comes up, but it wouldn't.

I guess it's hard to know without being in Mill's head. Though from what I've read it doesn't sound like he ever really wavered from favouring Britain having India as a colony.

I think this is an interesting analysis, but as others have indicated it could be better to frame this in terms of something like how these potential harms from saving human lives could be offset by donations to animal welfare charities, say.

Well, everyone will have their own emotional journey - not everyone with motivations to do good will have an experience like Mill's! But the point to not make improving social welfare the sole target and to have alternative sources of satisfaction seems to me quite common in discussions around EA and mental health, at least for those who do have difficulties.

I came across this extract from John Stuart Mill's autobiography on his experience of a period when he became depressed and lost motivation in his goal of improving society. It sounded similar to what I hear from time to time of EAs finding it difficult to maintain motivation and happiness alongside altruism, and thought some choice quotes would be interesting to share. Mill's solution was finding pleasure in other pursuits, particularly poetry.

Mill writes that his episode started in 1826, when he was 20 years old - but he had already been a keen utilitarian for 5 years and had been working for 3 years by this time, so was perhaps at a development point that not many would reach before they were into their early careers in the modern day.

From the winter of 1821, when I first read Bentham...I had what might truly be called an object in life; to be a reformer of the world. My conception of my own happiness was entirely identified with this object...This did very well for several years

But the time came when I awakened from this as from a dream. It was in the autumn of 1826. I was in a dull state of nerves...unsusceptible to enjoyment or pleasurable excitement...In this frame of mind it occurred to me to put the question directly to myself: "Suppose that all your objects in life were realized; that all the changes in institutions and opinions which you are looking forward to, could be completely effected at this very instant: would this be a great joy and happiness to you?" And an irrepressible self-consciousness distinctly answered, "No!" At this my heart sank within me: the whole foundation on which my life was constructed fell down...I seemed to have nothing left to live for.

For some months the cloud seemed to grow thicker and thicker...I became persuaded, that my love of mankind, and of excellence for its own sake, had worn itself out.

I frequently asked myself, if I could, or if I was bound to go on living, when life must be passed in this manner. I generally answered to myself that I did not think I could possibly bear it beyond a year. When, however, not more than half that duration of time had elapsed, a small ray of light broke in upon my gloom. I was reading, accidentally, Marmontel's Mémoires, and came to the passage which relates his father's death...A vivid conception of the scene and its feelings came over me, and I was moved to tears. From this moment my burden grew lighter.

I gradually found that the ordinary incidents of life could again give me some pleasure...and that there was, once more, excitement, though of a moderate, kind, in exerting myself for my opinions, and for the public good. Thus the cloud gradually drew off, and I again enjoyed life; and though I had several relapses, some of which lasted many months, I never again was as miserable as I had been.

The experiences of this period...led me to adopt a theory of life, very unlike that on which I had before I acted...Those only are happy (I thought) who have their minds fixed on some object other than their own happiness...followed not as a means, but as itself an ideal end.

This state of my thoughts and feelings made the fact of my reading Wordsworth for the first time (in the autumn of 1828), an important event of my life...[his poems] proved to be the precise thing for my mental wants at that particular juncture.

The result was that I gradually, but completely, emerged from my habitual depression, and was never again subject to it