Larks's Comments

Assumptions about the far future and cause priority
Interesting view. It seems to me like it makes sense, but I also feel like it'd be valuable for it to be fleshed out and critiqued further to see how solid it is. (Perhaps this has already been done somewhere - I do feel like I've heard vaguely similar arguments here and there.)

Thanks! I'm not sure if I made it up or not. I will try to find some time to write more about it.

Large EA House in Berkeley is looking for new housemates

Perhaps you might want to share your quarantine strategy? I can imagine that being important to people right now.

What are the key ongoing debates in EA?
I'm sure we can quibble about how the "Mental Health" should map to the "Psychedelics" category, though it seems clear that psychedelics are one of the most promising developments in mental health in the last few decades (breakthrough therapy designation from the FDA and all that).
If we assume half of the above considered psychedelics to be in the mental health bucket ...

This does not seem like a quibble to me at all. It seems 'clear' to you but this is by no means the case for most people. I would happily bet that well under half of those people were thinking psychedelics when they said mental health.

FHI Report: The Windfall Clause: Distributing the Benefits of AI for the Common Good
B.2. “The Windfall Clause will shift investment to competitive non-signatory firms.”

The concern here is that, when multiple firms are competing for windfall profits, a firm bound by the Clause will be at a competitive disadvantage because unbound firms could offer higher returns on new capital. That is, investors would prefer firms that are not subject to a “tax” on their profits in the form of the Windfall Clause. This is especially bad because it could mean that more prosocial firms (i.e., ones that have signed the Clause) would be at a disadvantage to non-signatory firms, making a prosocial “winner” of an AI development race less likely.238

This is a valid concern which warrants careful consideration. Our current best model for how to address this is that the Clause could commit (or at least allow for the option of) distributions of equity,* instead of cash. This could either take the form of stock options or contingent convertible bonds. This avoids the concern identified by allowing firms to, for example, issue new, preferred shares which would have superior claim to windfall profits compared to donees. This significantly diminishes the concern that the Clause would dilute the value of new shares issued in the company and allows the bound firm to raise capital unencumbered by debt owed under the Clause.† Notably, firm management would still have fiduciary duties towards stock-holding windfall donees.

I agree that the problem (that investors will prefer to invest in non-signatories, and hence it will reduce the likelihood of pro-social firms winning, if pro-social firms are more likely to sign) does seem like a credible issue. I found the description of the proposed solution rather confusing however. Given that I worked as an equity analyst for five years, I would be surprised if many other readers could understand it!

Here are my thoughts on a couple of possible versions of what you might be getting at- apologies if you actually intended something else altogether.

1) The clause will allow the company to make the required payments in stock rather than cash.

Unfortunately this doesn't really make much difference, because it is very easy for companies to alter this balance themselves. Consider that a company which had to make a $1 billion cash payment could fund this by issuing $1 billion worth of stock; conversely a company which had to issue stock to the fund could neutralise the effect on their share count by paying cash to buy back $1 billion worth of ordinary shares. This is the same reason why dividends are essentially identical to share buybacks.

2) The clause will allow subsequent financing to be raised that is senior to the windfall clause claim, and thus still attractive to investors.

'Senior' does not mean 'better' - it simply means that you have priority in the event of bankruptcy. However, the clause is already junior to all other obligations (because a bankrupt firm will be making ~0% of GDP in profit and hence have no clause obligations), so this doesn't really seem like it makes much difference. The issue is dilution in scenarios when the company does well, which is when the most junior claims (typically common equity, but in this case actually the clause) perform best.

The fundamental reason these two approaches will not work is that the value of an investment is determined by the net present value of future cashflows (and their probability distribution). Given that the clause is intended to have a fixed impact on these flows (as laid out in II.A.2), the impact on firm valuation is also rather fixed, and there is relatively little that clever financial engineering can do about it.

3) The clause will have claim only to profits attributable to the existing shares at the time of the signing on. Any subsequent equity will have a claim on profits unencumbered by the clause. For example, if a company with 80 shares signs on to the clause, then issues 10 more shares to the market, the maximum % of profits that would be owed is 50%*80/(80+10) = 44.4%

This would indeed avoid most of the problems in attracting new capital (save only the fear that a management team willing to screw over their previous investors will do so to you in the future, which is something investors think about a lot).

However, it would also largely undermine the clause by being easy to evade due to the fungibility of capital. Consider a new startup, founded by three guys in a basement, that signs the clause. Over the next few years they will raise many rounds of VC, eventually giving up the majority of the company, all excluded from the clause. Additionally, they pay themselves and employees in stock or stock options, which are also exempt from the clause. Eventually they IPO, having successfully diluted the clause-affected shares to ~1%. In order to finish the job, they then issue some additional new equity and use the proceeds to buy back the original shares.

One interesting point on the other side, however, is the curious tendency for tech investors to ignore dilution. Many companies will exclude stock-based-comp from their adjusted earnings, and analysts/investors are often willing to go along with this, saying "oh but it's a non-cash expense". Furthermore, SBC is excluded from Free Cash Flow, which is the preferred metric for many tech investors. So it is possible that (for a while) investors would simply ignore it.

FHI Report: The Windfall Clause: Distributing the Benefits of AI for the Common Good

Thanks very much for sharing this. It is nice to see some innovative thinking around AI governance.

I have a bunch of different thoughts, so I'll break them over multiple comments. This one mainly concerns the incentive effects.

> C.2. “The Windfall Clause operates like a progressive corporate income tax, and the ideal corporate income tax rate is 0%.”

> Some commentators argue that the ideal corporate tax rate is 0%. One common argument for this is that corporate income tax is not as progressive as its proponents think because corporate income is ultimately destined for shareholders, some of whom are wealthy, but many of whom are not. Better, then, to tax those wealthy shareholders more directly and let corporate profits flow less impeded to poorer ones. Additionally, current corporate taxes appear to burden both shareholders and, to a lesser extent, workers."

I think this is a bit of a strawman. While it is true that many people don't understand tax incidence and falsely assume the burden falls entirely on shareholders rather than workers and consumers, the main argument for the optimality of a 0% corporate tax rate is Chamley-Judd (see for example here) and related results. (There are some informal descriptions of the result here and here.) The argument is about disincentives to invest reducing long-run growth and thereby making everyone poorer, not a short-term distributional effect. (The standard counter-argument to Chamley Judd, as far as I know, is to effectively apply lots of temporal discounting, but this is not available to longtermist EAs).

This is sort of covered in B.1., but I do not think the responses are very persuasive. The main response is rather glib:

Further, by capping firm obligations at 50% of marginal profits, the Clause leaves room for innovation to be invested in even at incredibly high profit levels.231

There are a lot desirable investments which would be rendered uneconomic. The fact that some investment will continue at a reduced level does not mean that missing out on the other forgone projects is not a great cost! For example, a 20% pre-tax return on investment for a moderately risky project is highly attractive - but after ~25% corporate taxes and ~50% windfall clause, this is a mere 5% return* - almost definitely below their cost of capital, and hence society will probably miss out on the benefits. Citation 231, which seems like it should be doing most of the work here, instead references a passing comment in a pop-sci book about individual taxes:

There's also an argument that a big part of the very high earnings of many 'superstars' are also rents. These questions turn on whether most professional athletes, CEOs, media personalities, or rock stars are genuinely motivated by the absolute level of their compensation verses the relative compensation, their fame, or their intrinsic love of their work.

But corporations are much less motivated by fame and love of their work than individuals, so this does not seem very relevant, and furthermore it does not address the inter-temporal issue which is the main objection to corporation taxes.

I also think the sub-responses are unsatisfying. You mention that the clause will be voluntary:

> Firstly, we expect firms to agree to the Clause only if it is largely in their self-interest

But this does not mean it won't reduce incentives to innovate. Firms can rationally take actions that reduce their future innovation (e.g. selling off an innovative but risky division for a good price). A firm might voluntarily sign up now, when the expected cost is low, but then see their incentives dramatically curtailed later, when the cost is large. Furthermore, firms can voluntarily but irrationally reduce their incentives to innovate - for example a CEO might sign up for the clause because he personally got a lot of positive press for doing so, even at the cost of the firm.

Additionally, by publicising this idea you are changing the landscape - a firm which might have seen no reason to sign up might now feel pressured to do so after a public campaign, even though their submission is 'voluntary'.

The report then goes on to discuss externalities:

> Secondly, unbridled incentives to innovate are not necessarily always good, particularly when many of the potential downsides of that innovation are externalized in the form of public harms. The Windfall Clause attempts to internalize some of these externalities to the signatory, which hopefully contributes to steering innovation incentives in ways that minimize these negative externalities and compensate their bearers.

Here you approvingly cite Seb's paper, but I do not think it supports your point at all. Firms have both positive and negative externalities, and causing them to internalise them requires tailored solutions - e.g. a carbon tax. 'Being very profitable' is not a negative externality, so a tax on profits is not an effective way of minimising negative externalities. Similarly, the Malicious Use paper is mainly about specific bad use cases, rather than size qua size being undesirable. Moreover, size has little to do with Seb's argument, which is about estimating the costs of specific research proposals when applying for grants.

Finally, one must consider that under windfall scenarios the gains from innovation are already substantial, suggesting that globally it is more important to focus on distribution of gains than incentivizing additional innovation.

I strongly disagree with this non-sequitur. The fact that we have achieved some level of material success now doesn't mean that the future opportunity isn't very large. Again, Chamley-Judd is the classic result in the space, suggesting that it is never appropriate to tax investment for distributional purposes - if the latter must be done, it should be done with individual-level consumption/income taxation. This should be especially clear to EAs who are aware of the astronomical waste of potentially forgoing or delaying growth.

Elsewhere in the document you do hint at another response - namely that by adopting the clause, companies will help avoid future taxation (though I am sceptical):

> A Windfall Clause could build goodwill among the public, dampening harmful public antagonism for a small (expected) cost. Governments may be less likely to excessively tax or expropriate firms committed to providing a public good through the Windfall Clause.


> However, from a public and employee relations perspective, the Clause may be more appealing than taxation because the Clause is a cooperative, proactive, and supererogatory action. So, to the extent that the Windfall Clause merely replaces taxation, the Windfall Clause confers reputational benefits onto the signatory at no additional cos

However, it seems that the document equivocates on whether or not the clause is to reduce taxes, as elsewhere in the document you deny this:

> the Windfall Clause is not intended to be a substitute for taxation schemes. We also note that, as a private contract, the Windfall Clause cannot supersede taxation. Thus, if a state wants to tax the windfall, the Clause is not intended to stop it. Indeed, taxation efforts that broadly align with the goals and design principles of the Windfall Clause are highly desirable

\* for clarity of exposition I am assuming the donation is not tax deductible, but the point is not dramatically altered if it is.

Short-Term AI Alignment as a Priority Cause

Many of these advantages (e.g. aligned recommenders pushing people towards longtermism, or animal rights) seem more like aligning recommenders with our values than any neutral account of alignment. It seems than any ideology could similarly claim that aligned recommenders are important for introducing people to libertarianism/socialism/conservatism/feminism etc. In contrast, this probably isn't the best interests of the viewer - e.g. your average omnivore probably doesn't want to be recommended videos about animal suffering.

Do impact certificates help if you're not sure your work is effective?
I'm concerned that splitting the "vote" between these two methods will do harm to the community's ability to decide what types of work are good.

Could you go into detail about why you think this would be bad? Typically when you are uncertain about something it is good to have multiple (semi-) independent indicators, as you can get a more accurate overall impression by combining the two in some way.

Request for Feedback: Draft of a COI policy for the Long Term Future Fund

Thanks for writing this up for comment.

Quick possible oversight - I didn't see any discussion of recusal because the fund member is employed or receives funds from the potential grantee? Sorry if I just misread! The closest I saw was this:

A very close friend or partner of a fund member is employed, receiving funds from, or has some kind of other directly dependent relationship to the potential grantee

Right now I assume this would mainly apply to you (CFAR) and possibly Alex.

Separately, you mentioned OpenPhil's policy of (non-) disclosure as an example to emulate. I strongly disagree with this, for two reasons.

Firstly, I think OpenPhil's policy is bad. They enacted this policy as part of their general movement towards secrecy, but the actual reasons they described for sharing less details about their evaluation (i.e. issues are too complex, don't want to aid hostile actors etc.) do not seem to be that relevant to not disclosing conflicts. Certainly, OpenPhil's policy of non-disclosure makes me trust their work significantly less now as I have to assume there is a significant chance any given decision was unfairly biased.

Secondly, there are significant differences between OpenPhil and you guys. In particular, OpenPhil's main job is advising a very small number of individuals, who (I presume) have access to many private details that they do not need to make public. Additionally, those individuals have a considerable amount of influence over OpenPhil. In the case of the LTFF, however, donors are reliant on public disclosure in order to be able to evaluate the fund. It is like the difference between a private company (who have almost no public disclosure requirements) and a public one (who have a lot of disclosure requirements).

Load More