An independent researcher of ethics, AI safety, and AI impacts. LessWrong: https://www.lesswrong.com/users/roman-leventov. Twitter: https://twitter.com/leventov. E-mail: firstname.lastname@example.org (the preferred mode of communication).
Alien valuesMaximalist desire for world dominationConvergence to a utility functionVery competent strategizing, of the “treacherous turn” varietySelf-improvement
Alien values are guaranteed unless we explicitly impart non-alien ethics to AI, which we currently don't know how to do, and don't know (or can't agree) what that ethics should be like. Next two points are synonyms and are also basically synonyms to "alien values". The treacherous turn is indeed unlikely (link).
Self-improvement is given, the only question is where is the "ceiling" of this improvement. It might not be that "far", by some measure, from human intelligence, or that difference may still not allow AI to plan that far ahead due to the intrinsic unpredictability of the world. So the world may start to move extremely fast (see below), but the horizon of planning and predictability of that movement may not be longer than it is now (or it could be even shorter).
For a given operationalization of AGI, e.g., good enough to be forecasted on, I think that there is some possibility that we will reach such a level of capabilities, and yet that this will not be very impressive or world-changing, even if it would have looked like magic to previous generations. More specifically, it seems plausible that AI will continue to improve without soon reaching high shock levels which exceed humanity’s ability to adapt.
I think you implicitly underestimate the cost of coordination among humans. Huge corporations are powerful but also very slow to act. AI corporations will be very powerful and also very fast and potentially very coherent in their strategy. This will be a massive change.
It could be morally good to return some grants if there is a good theory on how this will lead to better results for the people involved and their families, for the communities they are part of (primarily, the EA community), for society, and for the civilisation.
Some deontological motives and considerations could be a part of such a theory. For example, as other people already mentioned in this discussion, returning grants could send a valuable signal to the EA community and to society.
However, it seems to me, the framing of the question "Under what conditions should FTX grantees voluntarily return their grants?" hints at the possibility of some hard-and-fast deontological algorithm for deciding when grants should be returned. I don't think such an algorithm exists. The theories for why returning funds would be good should be far more nuanced, and applicable to very narrow strata of grantees and victims respectively (perhaps even down to individual grantees and individual victims), rather than large strata such as "all grantees" and "all victims", or even "1% of victims who were affected the most in terms of the portion of their net worth that was destroyed".
Considering the above, I think just returning money to FTXFF (or another pool of money) would be ineffective. And even creating a short-lived organisation to administer claims for returns from the victims will be ineffective, too (especially considering the opportunity cost for people who can quickly create locally-effective organisations of this sort: I believe such people have much more valuable things to organise, from the EA perspective).
I think a solution that could be low-investment and also relatively effective is organising a forum where individual victims share their stories and ask for help, and individual grantees can come and respond, assessing their own situation and the situation of the victim, that is, building "a good theory". And then publicising this forum among both the victims and the grantees. This also doesn't mean grantees should return their entire grants, they may help a little, according to their situation and the situation of the particular victim. However, one complication with this solution might be: how could the stories of the victims be verified?
In this setup, grantees should also consider the implications of their decisions for the community and society, not just themselves and the victim. While the latter are highly individual, the former are mostly shared. So it would make sense for some people who are experts in community strategy, sociology, and ethics to write a few essays on this topic that grantees would be advised to read before visiting the forum. (I'm not such an expert.) Of course, individual grantees would still be free to form their own sub-theory regarding these "high-level implications", according to their own understanding of the community strategy and ethics.