All of A_donor's Comments + Replies

Impact Markets: The Annoying Details

Yes, and greater GDP maps fairly well to greater effectiveness of altruism. I think you're focused on downside risks too strongly. They exist, and they are worth mitigating, but inaction due to fear of them will cause far more harm. Inaction due to heckler's veto is a not a free outcome.

Companies not being loss-limited would not cause them to stop producing x-risks when the literal death of all their humans is an insufficient motivation to discourage them. It would reduce a bunch of other categories of harm, but we've converged to accepting that risk to avoid crippling risk aversion in the economy.

Impact Markets: The Annoying Details
  1. Seems essentially fine. There's a reason society converged to loss-limited companies being the right thing to do, even though there is unlimited gain and limited downside, and that's that individuals tend to be far too risk averse. Exposing them to a risk to the rest of their portfolio should be more than sufficient to make this not a concern.
  2. Might be a fair point, but remember, this is in the case where some project was predictably net negative and then actually was badly net negative. My guess is at least some funders would be willing to step in and disincentivise that kind of activity, and the threat of it would keep people off the worst projects.
1ofer20d
I think the reason that states tend to allow loss-limited companies is that it causes them to have larger GDP (and thus all the good/adaptive things that are caused by having larger GDP). But loss-limited companies may be a bad thing from an EA perspective, considering that such companies may be financially incentivized to act in net-negative ways (e.g. exacerbating x-risks), especially in situations where lawmakers/regulators are lagging behind.
Impact Markets: The Annoying Details

If the distribution mismatch problem is not mitigated (and it seems hard to mitigate), investors are incentivized to fund high-stakes projects while regarding potential harmful outcomes as if they were neutral.

I thought a bunch more about this, and I do think there is something here worth paying attention to.

I am not certain that there is a notable enough pool of the projects in the category you're worried about to offset the benefits of impact markets, but we would incentivise those that exist, and that has a cost.

If we're limited to accredited investors,... (read more)

3ofer22d
I think this approach has the following problems: 1. Investors will still be risking only the total amount of money they invest in the market (or place as a collateral), while their potential gain is unlimited. 2. People tend to avoid doing things that directly financially harm other individuals. Therefore, I expect retro funders would usually not use their power to mark a project as "ex-ante net negative", even if it was a free action and the project was clearly ex-ante net negative (let alone if the retro funders need to spend money on doing it; and if it's very hard to judge whether the project was ex-ante net negative, which seems a much more common situation).
Impact Markets: The Annoying Details

I should have phrased differently: It's not that hard to pick out highly risky projects retroactively, relative to identifying them prospectively. I also think that the reference class which is most worrying is genuinely not that hard to identify as strongly negative EV.

Impact markets don't solve the problem of funders being able to fund harmful projects. But they don't make it differentially worse (it empowers funders generally, but I don't expect you would argue that grantmakers are net negative, so this still comes out net-positive).

I would welcome atte... (read more)

3ofer24d
Do you mean that, if a project ends up being harmful we have Bayesian evidence that it was ex-ante highly risky? If so, I agree. But that fact does not alleviate the distribution mismatch problem [https://forum.effectivealtruism.org/posts/74rz7b8fztCsKotL6/impact-markets-may-incentivize-predictably-net-negative#The_risk] , which is caused by the prospect of a risky project ending up going well. If the distribution mismatch problem is not mitigated (and it seems hard to mitigate [https://forum.effectivealtruism.org/posts/74rz7b8fztCsKotL6/impact-markets-may-incentivize-predictably-net-negative#Mitigating_the_risk_is_hard] ), investors are incentivized to fund high-stakes projects while regarding potential harmful outcomes as if they were neutral. (Including in anthropogenic x-risks and meta-EA domains.) That is not the case with EA funders today. I think this is a highly over-optimistic take about cranking up the profit-seeking lever in EA and the ability to mitigate the effects of Goodhart's law. It seems that when humans have an opportunity to make a lot of money (without breaking laws or norms) at the expense of some altruistic values, they usually behave in a way that is aligned with their local incentives (while convincing themselves it's also the altruistic thing to do). If you run a fully controlled (Web2) impact market for 6-12 months, and the market funds great projects/posts and there's no sign of trouble, will you then launch a decentralized impact market that no one can control (in which people can sell the impact of recruiting additional retro funders, and the impact of establishing that very market)?
Impact Markets: The Annoying Details

I think these concerns are largely incorrect, and avoiding implementing this technology due to them would be a tragedy. Here's my reasoning:

  1. It's not that hard to see when a project was at risk of having large downsides, oraculars can avoid retrofunding these just as they would avoid normally funding them.[1]
  2. If you assume oraculars who are fine with incentivising doing large amounts of expected harm in order to get credit for something if it goes well (which I think is a flawed assumption), well.. they can do that with normal funding. Retrofunding does not
... (read more)

It's not that hard to see when a project was at risk of having large downsides

I strongly disagree. It's often extremely hard to judge whether a project related to anthropogenic x-risks was ex-ante net-negative. For example, was the creation of OpenAI/Anthropic/CSET net-positive or net-negative (ex-ante)? How about any particular gain-of-function research effort, or the creation of any particular BSL-4 virology lab?

Given a past project that is related to anthropogenic x-risks or meta-EA, it can be extremely hard to evaluate the ex-ante potential harm tha... (read more)

Impact Markets: The Annoying Details

Quick opinions on the choice branches:

1: What Is The Basic Format Of The Market?

Custom: Awesome Auto Auction, but closest to D: Fractionalized Impact Shares With Assurance Contract.

2: What Is Being Sold?

B: Credit For Funding The Project - Funding a project retroactively should not be considered morally different from funding it in any other way, creating a novel kind of moral accounting seems overcomplicating things and hard to get actual consensus / common knowledge on.

3: How Should The Market Handle Projects With Variable Funding Needs?

Custom: Awesome Au... (read more)

Impact Markets: The Annoying Details

I did a bunch of mechanism design on the details of #1 alongside the Impact Markets people!

Key desiderata:

  • Low overhead for participants in the market - Incentivising these people to spend lots of brainpower trying to work out when you buy and sell is massively suboptimal. You want a system that is buy and forget and just works.
  • Minimize hard feelings around selling your cert at a loss or seeing equivalent ones go for a very low price - Related to your #II - C
  • Impact authors automatically get some fraction of the rewards of a very successful project[1]
  • Ability
... (read more)
MIRI Conversations: Technology Forecasting & Gradualism (Distillation)

This is great! Ties the core threads of thought around this disagreement in a cohesive and well-referenced way, much more digestible than the originals.

I'd suggest cross-posting to LessWrong, and including links to the relevant posts on LW/EAF so that the "Mentioned in" feature of this forum generates backlinks to your work from those posts.

1Callum McDougall1mo
Ah yep, I'd been planning to do that but had forgotten, will do now. Thanks!
EA for dumb people?

The Parable of the Talents, especially the part starting at:

But I think the situation can also be somewhat rosier than that.

Ozy once told me that the law of comparative advantage was one of the most inspirational things they had ever read. This was sufficiently strange that I demanded an explanation.

Ozy said that it proves everyone can contribute. Even if you are worse than everyone else at everything, you can still participate in global trade and other people will pay you money. It may not be very much money, but it will be some, and it will be a measure

... (read more)

I also want to chime in here and say that it was a bit of a shock for me coming into the EA community also: I was one of the more analytical people in most of my friendship groups, yet it was pretty quickly clear to me that my comparative advantage in this community was actually EQ, communications, and management. I'm glad to work with some incredibly smart analytical people who are kind enough to (a) help me understand things that confuse me when I'm frank about what I don't understand; and (b) remind me what else I bring to the table.

Another relevant Slate Star Codex post is Against Individual IQ Worries.

What Are Your Software Needs?

Talk to HaonChan and Jehan on the Bountied Rationality Discord. They're trying to build this.

Affectable universe?

Not sure if it's the widely used definition, but I think of affectable as anything in our future light cone, with accessible as just the bits which we could get something (e.g. a von Neumann probe) physically to before the expansion of the universe takes it away from our reach, which is a smaller bubble because we can't send matter at light speed, our probe would have to slow down at the other end, and the expansion of the universe robs our probe of velocity relative to distant galaxies over very long periods.

Edit: The FHI has a relevant paper: https://www... (read more)

The Vultures Are Circling

I was considering writing something like this up a a while back, but didn't have enough evidence directly, was mostly working of too few examples as a grantmaker and general models. Glad this concern is being broadcast.

I did come up with a proposal for addressing parts of the problem over on the "Please pitch ideas to potential EA CTOs" post. If you're a software dev who wants to help build a tool which might make the vultures less able to eat at least parts of EA please read over the proposal and ping me if interested.

Please pitch ideas to potential EA CTOs

Additional layer: Have the researchers have a separate "Which infrastructure / support has been most valuable to you?" category of karma, and use that to help direct funding towards the most valuable parts of the infrastructure ecosystem to support alignment. This should be one way, with researchers able to send this toward infrastructure, but not the reverse since research is the goal.

Preserving and continuing alignment research through a severe global catastrophe

The purpose of preserving alignment is not to get back to AI as quickly as possible, but to make it more likely that when we eventually do climb the tech tree we are more likely to be able to align advanced AIs. Even if we have to reinvent a large number of technologies, having alignment research ready represents a (slightly non-standard) form of differential technological development rather than simply speeding up the recovery overall.

-2acylhalide5mo
Makes sense. That seems like a significantly more complex task though. There's firstly the moral question of how you trade-off between P(recovery) for each tech level, and P(aligned AI | recovery). And then there's practical considerations on how much providing or not providing different technologies can impact these probabilities. I do wish you luck in your project if you feel it is sufficiently urgent. But I also feel that eventually such questions need to be considered, and that coordinated action by EAs will be useful.
Preserving and continuing alignment research through a severe global catastrophe

Agreed that civilization restart manuals would be good, would be happy to have the alignment archives stored alongside those. Would prefer not to hold up getting a MVP of this much smaller and easier archive in place waiting for that to come together though.

1acylhalide5mo
I see. Why though? A civilisation starting from scratch needs to rebuild hundreds of technologies before it has computers or AI.
Preserving and continuing alignment research through a severe global catastrophe

My guess is these are great for longevity,  but maybe prohibitively expensive[1] if you want to print out e.g. the entire alignment forum plus other papers. 

Could be good for a smaller selected key insights collection, if that exists somewhere?

  1. ^

    Likely reference class is gravestones. I'm getting numbers like:  "Extra characters are approximately $10 thereafter" and "It costs around £1.95 per letter or character", even with a bulk discount that's going to add up.

3Greg_Colbourn5mo
Now I'm imagining a friendly AGI etching the Textbook From The Future [https://www.lesswrong.com/posts/34Gkqus9vusXRevR8/late-2021-miri-conversations-ama-discussion?commentId=DBmPemwANLoBzwc8Z#DBmPemwANLoBzwc8Z] on stone tablets. But it would be an interesting exercise to try and condense the key insights made to date into 1k or 10k characters.
Please pitch ideas to potential EA CTOs

AI alignment is rapidly scaling funding, and this means grantmakers will be stretched thin and less able to reliably avoid giving money to people who are not mission aligned, not producing useful research, or worst of all just want to extract money from the system. This has the potential to cause massive problems down the road, both by producing distracting low-quality research and by setting up expectations which will cause drama if someone is defunded later on while there's still a lot of money flowing to others.

An attack-resistant EigenKarma[1]-like net... (read more)

1A_donor3mo
Additional layer: Have the researchers have a separate "Which infrastructure / support has been most valuable to you?" category of karma, and use that to help direct funding towards the most valuable parts of the infrastructure ecosystem to support alignment. This should be one way, with researchers able to send this toward infrastructure, but not the reverse since research is the goal.
2Greg_Colbourn5mo
Exit strategy: sell to Reddit? (Although to be fair I guess there won't be much in the way of protected IP)
The Future Fund’s Regranting Program

Yes: "Regrantors will be compensated for their work based on the quality and volume of their grantmaking."

The Future Fund’s Regranting Program

I'd be very interested  in joining as a regranter, though it may make sense to wait a few years, by which point I will have donated most of my crypto pool and gained a bunch of experience. You can see my current strategy at Being an individual alignment grantmaker.

Edit: Does screening for conflicts of interest mean not allowing regranters to grant to people they know? If yes, I can see the reasoning, but if I was operating under this rule it would have blocked several of my most promising grants, which I found through personal connections. I would pro... (read more)

2leopold5mo
You are welcome to apply now! Regrantors are able to make grants to people they know (in fact, having a diverse network is part of what makes for an effective regrantor); they just have to disclose if there's a conflict of interest, and we may reject a grant if we don't feel comfortable with it on those grounds. We don't currently have a network for regrantors that is open for external people to join.