Paul_Christiano

Comments

Dutch anti-trust regulator bans pro-animal welfare chicken cartel

Is your impression that if customers were willing to pay for it, then that wouldn't be sufficient cause to say that it benefited customers? (Does that mean that e.g. a standard ensuring that children's food doesn't cause discomfort also can't be protected, since it benefits customers' kids rather than customers themselves?)

Dutch anti-trust regulator bans pro-animal welfare chicken cartel

These cases are also interesting for alignment agreements between AI labs, and it's interesting to see it playing out in practice. Cullen wrote about this here much better than I will.

Roughly speaking, if individual consumers would prefer use a riskier AI (because costs are externalized) then it seems like an agreement to make AI safer-but-more-expensive would run afoul of the same principles as this chicken-welfare agreement.

On paper, there are some reasons that the AI alignment case should be easier than the chicken-welfare case: (i) using unsafe AI hurts non-customer humans, and AI customers care more about other humans than they do about chickens, (ii) deploying unaligned AI actually likely hurts other AI customers in particular (since they will be the main ones competing with the unaligned but more sophisticated AI). So it's likely that every individual AI customer would  benefit.

Unfortunately, it seems like the same thing could be true in the chicken case---every individual customer could prefer the world with the welfare agreement---and it wouldn't change the regulator's decision.

For example, suppose that Dutch consumers eat 100 million chickens a year, 10/year for each of 10 million customers. Customer surveys discover that customers would only be willing to pay $0.01 for a chicken to have more space and a slightly longer life, but that these reforms increase chicken prices by $1. So they strike down the reform.

But with welfare standards in place, each customer pays an extra $10/year for chicken and 100 million chickens have improved lives, with a cost per chicken of less than $0.0000001/chicken, thousands of times lower than their WTP. (This is the same dynamic described here.) So every chicken consumer prefers the world where the standards are in place, despite not being willing to pay money to improve the lives of the tiny number of chickens they eat personally. This seems to be a very common reaction to discussions of animal welfare ("what difference does my consumption make? I can't change the way most chickens are treated...")

Because the number of chicken-eaters is so large, the relevant question in the survey should be "Would you prefer that someone else pay $X in order to improve chicken welfare?", making a tradeoff between two strangers. That's the relevant question for them, since the welfare standards mostly affect other people.

Analogously, if you ask AI consumers "Would you prefer have an aligned AI, or a slightly more sophisticated unaligned AI?" they could easily all say "I want the more sophisticated one," even if every single human would be better off if there were an agreement to make only aligned AI. If an anti-trust regulator used the same standard as in this case, it seems like they would throw out an alignment agreement because of that, even knowing that it would make every single human worse off.

I still think in practice AI alignment agreements would be fine for a variety of reasons. For example, I think if you ran a customer survey it's likely people would say they prefer use aligned AI even if it would disadvantage them personally because public sentiment towards AI is very different and the regulatory impulse is stronger. (Though I find it hard to believe that anything would end up hinging on such a survey, and even more strongly I think it would never come to this because there would be much less political pressure to enforce anti-trust.)

Alternatives to donor lotteries

I guess I wouldn't recommend the donor lottery to people who wouldn't be happy entering a regular lottery for their charitable giving

Strong +1.

If I won a donor lottery, I would consider myself to have no obligation whatsoever towards the other lottery participants, and I think many other lottery participants feel the same way. So it's potentially quite bad if some participants are thinking of me  as an "allocator" of their money. To the extent there is ambiguity in the current setup, it seems important to try to eliminate that.

[Link post] Are we approaching the singularity?
  1. I think that acceleration is autocorrelated---if things are accelerating rapidly at time T they are also more likely to be accelerating rapidly at time T+1. That's intuitively pretty likely, and it seems to show up pretty strongly in the data. Roodman makes no attempt to model it, in the interest of simplicity and analytical tractability. We are currently in a stagnant period, and so I think you should expect continuing stagnation. I'm not sure exactly how large the effect (and obviously it depends on the model) is but I think it's at least a 20-40 year delay. (There are two related angles to get a sense for the effect: one is to observe that autocorrelations seem to fade away on the timescale of a few doublings, rather than being driven by some amount of calendar time, and the other is to just look at the fact that we've had something like ~40 years of relative stagnation.)
  2. I think it's plausible that historical acceleration is driven by population growth, and that just won't really happen going forward. So at a minimum we should be uncertain betwe3en roodman's model and one that separates out population explicitly, which will tend to stagnate around the time population is limited by fertility rather than productivity.

(I agree with Max Daniel below that I don't think that Nordhaus' methodology is inherently more trustworthy. I think it's dealing with a relatively small amount of pretty short-term data, and is generally using a much more opinionated model of what technological change would look like.)

[Link post] Are we approaching the singularity?

The relevant section is VII. Summarizing the six empirical tests:

  1. You'd expect productivity growth to accelerate as you approach the singularity, but it is slowing.
  2. The capital share should approach 100% as you approach the singularity. The share is growing, but at the slow rate of ~0.5%/year. At that rate it would take roughly 100 years to approach 100%.
  3. Capital should get very cheap as you approach the singularity. But capital costs (outside of computers) are falling relatively slowly.
  4. The total stock of capital should get large as you approach the singularity. In fact the stock of capital is slowly falling relative to output.
  5. Information should become an increasingly important part of the capital stock as you approach the singularity. This share is increasing, but will also take >100 years to become dominant.
  6. Wage grow should accelerate as you approach the singularity, but it is slowing.

I would group these into two basic classes of evidence:

  • We aren't getting much more productive, but that's what a singularity is supposed to be all about.
  • Capital and IT extrapolations are potentially compatible with a singularity, but only a timescale of 100+ years.

I'd agree that these seem like two points of evidence against singularity-soon, and I think that if I were going on outside-view economic arguments I'd probably be <50% singularity by 2100. (Though I'd still have a meaningful probability soon, and even at 100 years the prospect of a singularity would be one of the most important facts about the basic shape of the future.)

There are some more detailed aspects of the model that I don't buy, e.g. the very high share of information capital and persistent slow growth of physical capital. But I don't think they really affect the bottom line.

Three Impacts of Machine Intelligence

If the market can't price 30-year cashflows, it can't price anything, since for any infinitely-lived asset (eg stocks!), most of the present-discounted value of future cash flows is far in the future. 

If an asset pays me far in the future,  then long-term interest rates are one factor affecting its price. But it seems to me that in most cases that factor still explains a minority of variation in prices (and because it's a slowly-varying factor it's quite hard to make money by predicting it).

For example, there is a ton of uncertainty about how much money any given company is going to make next year. We get frequent feedback signals about those predictions, and people who win bets on them immediately get returns that let them show how good they are and invest more, and so that's the kind of case where I'd be more scared of outpredicting the market.

So I guess that's saying that I expect the relative prices of stocks to be much more efficient than the absolute level.

See eg this Ralph Koijen thread and linked paper, "the first 10 years of dividends only make up ~20% of the value of the stock market. 80% is due to value of cash flows beyond 10 years"

Haven't looked at the claim but it looks kind of misleading. Dividend yield for SPY is <2% which I guess is what they are talking about? But buyback yield is a further 3%, and with a 5% yield you're getting 40% of the value in the first 10 years, which sounds more like it. So that would mean that you've gotten half of the value within 13.5 years instead of 31 years.

Technically the stock is still valued based on the future dividends, and a buyback is just decreasing outstanding shares and so increasing earnings per share. But for the purpose of pricing the stock it should make no difference whether earnings are distributed as dividends or buybacks, so the fact that buybacks push cashflows to the future can't possibly affect the difficulty of pricing stocks.

Put a different way, the value of a buyback to investors doesn't depend on the actual size of future cashflows, nor on the discount rate. Those are both cancelled out because they are factored into the price at which the company is able to buy back its shares. (E.g. if PepsiCo was making all of its earnings in the next 5 years, and ploughing them into buybacks, after which they made a steady stream of not-much-money, then PepsiCo prices would still be equal to the NPV of dividends, but the current PepsiCo price would just be an estimate of earnings over the next 5 years and would have almost no relationship to long-term interest rates.)

Even if this is right it doesn't affect your overall point too much though, since 10-20 year time horizons are practically as bad as 30-60 year time horizons.

Three Impacts of Machine Intelligence

I I think the market just doesn't put much probability on a crazy AI boom anytime soon. If you expect such a boom then there are plenty of bets you probably want to make. (I am personally short US 30-year debt, though it's a very small part of my AI-boom portfolio.)

I think it's very hard for the market to get 30-year debt prices right because the time horizons are so long and they depend on super hard empirical questions with ~0 feedback. Prices are also determined by supply and demand across a truly huge number of traders, and making this trade locks up your money forever and can't be leveraged too much. So market forecasts are basically just a reflection of broad intellectual consensus about the future of growth (rather than views of the "smart money" or anything), and the mispricing is just a restatement of the fact that AI-boom is a contrarian position.

AGB's Shortform

Some scattered thoughts (sorry for such a long comment!). Organized in order rather than by importance---I think the most important argument for me is the analogy to computers.

  • It's possible to write "Humanity survives the next billion years" as a conjunction of a billion events (humanity survives year 1, and year 2, and...). It's also possible to write "humanity goes extinct next year" as a conjunction of a billion events (Alice dies, and Bob dies, and...). Both of those are quite weak prima facie justifications for assigning high confidence. You could say that the second conjunction is different, because the billionth person is very likely to die once the others have died (since there has apparently been some kind of catastrophe), but the same is true for survival. In both cases there are conceivable events that would cause every term of the conjunction to be true,  and we need to address the probability of those common causes directly. Being able to write the claim as a conjunction doesn't seem to help you get to extreme probabilities without an argument about independence.
  • I feel you should be very hesitant to assign 99%+ probabilities without a good argument, and I don't think this is about anchoring to percent. The burden of proof gets stronger and stronger as you move closer to 1, and 100 is getting to be a big number. I think this is less likely to be a tractable disagreement than the other bullets but it seems worth mentioning for completeness. I'm curious if  you think there are other natural statements where the kind of heuristic you are describing (or any other similarly abstract heuristic) would justifiably get you to such high confidences. I agree with Max Daniel's point that it doesn't work for realistic versions of claims like "This coin will come up heads 30 times in a row." You say that it's not exclusive to simplified models but I think I'd be similarly skeptical of any application of this principle. (More generally, I think it's not surprising to assign very small probabilities to complex statements based on weak evidence, but that it will happen much more rarely for simple statements. It doesn't seem promising to get into that though.)
  • I think space colonization is probably possible, though getting up to probabilities like 50% for space colonization feasibility would be a much longer discussion. (I personally think >50% probability is much more reasonable than <10%.) If there is a significant probability that we colonize space, and that spreading out makes the survival of different colonists independent (as it appears it would), then it seems like we end up with some significant probability of survival. That said, I would also assign ~1/2 probability to surviving a billion years even if we were confined to Earth. I could imagine being argued down to 1/4 or even 1/8 but each successive factor of 2 seems much harder. So in some sense the disagreement isn't really about colonization.
  • Stepping back, I think the key object-level questions are something like "Is there any way to build a civilization that is very stable?" and "Will people try?" It seems to me you should have a fairly high probability on "yes" to both questions. I don't think you have to invoke super-aligned AI to justify that conclusion---it's easy to imagine organizing society in a way which drives existing extinction risks to negligible levels, and once that's done it's not clear where you'd get to 90%+ probabilities for new risks emerging that are much harder to reduce. (I'm not sure which step of this you get off the boat for---is it that you can't imagine a world that say reduced the risk of an engineered pandemic killing everyone to < 1/billion per year? Or that you think it's very likely other much harder-to-reduce risks would emerge?)
  • A lot of this is about burden of proof arguments. Is the burden of proof on someone to exhibit a risk that's very hard to reduce, or someone to argue that there exists no risk that is hard to reduce? Once we're talking about 10% or 1% probabilities it seems clear to me that the burden of proof is on the confident person. You could try to say "The claim of 'no bad risks' is a conjunction over all possible risks, so it's pretty unlikely" but I could just as well say "The claim about 'the risk is irreducible' is a conjunction over all possible reduction strategies, so it's pretty unlikely" so I don't think this gets us out of the stalemate (and the stalemate is plenty to justify uncertainty).
  • I do furthermore think that we can discuss concrete (kind of crazy) civilizations that are likely to have negligible levels of risk, given that e.g. (i) we have existence proofs for highly reliable machines over billion-year timescales, namely life, (ii) we have existence proofs for computers if you can build reliable machinery of any kind, (iii) it's easy to construct programs that appear to be morally relevant but which would manifestly keep running indefinitely.  We can't get too far with this kind of concrete argument, since any particular future we can imagine is bound to be pretty unlikely. But it's relevant to me that e.g. stable-civilization scenarios seem about as gut-level plausible to me as non-AI extinction scenarios do in the 21st century.
  • Consider the analogous question "Is it possible to build computers that successfully carry out trillions of operations without errors that corrupt the final result?" My understanding is that in the early 20th century this question was seriously debated (though that's not important to my point), and it feels very similar to your question. It's very easy for a computational error to cascade and change the final result of a computation. It's possible to take various precautions to reduce the probability of an uncorrected error, but why think that it's possible to reduce that risk to levels lower than 1 in a trillion, given that all observed computers have had fairly high error rates? Moreover, it seems that error rates are growing  as we build bigger and bigger computers, since each element has an independent failure rate, including the machinery designed to correct errors. To really settle this we need to get into engineering details, but until you've gotten into those details I think it's clearly unwise to assign very low probability to building a computer that carries out trillions of steps successfully---the space of possible designs is large and people are going to try to find one that works, so you'd need to have some good argument about why to be confident that they are going to fail.
  • You could say that computers are an exceptional example I've chosen with hindsight. But I'm left wondering if there are any valid applications of this kind of heuristic--what's the reference class of which "highly reliable computers" are exceptional rather than typical?
  • If someone said:"A billion years is a long time. Any given thing that can plausibly happen should probably be expected to happen over that time period" then I'd ask about why life survived the last billion years.
  • You could say that "a billion years" is a really long time for human civilization (given that important changes tend to happen within decades or centuries) but not a long time for intelligent life (given that important changes takes millions of years). This is similar to what happens if you appeal to current levels of extinction risk being really high. I don't buy this because life on earth is currently at a period of unprecedentedly rapid change. You should have some reasonable probability of returning to more historically typical timescales of hundreds of millions of years, which in turn gives you a reasonable overall probability on surviving for hundreds of millions of years. (Actually I think we should have >50% probabilities for reversion to lower timescales, since we can tell that the current period of rapid growth will soon be over. Over our history rapid change and rapid growth have basically coincided, so it's particularly plausible that returning to slow-growth will also return to slow-change.)
  • Applying the rule of thumb for estimating lifetimes to "the human species" rather than "intelligent life" seems like it's doing a huge amount of work. It might be reasonable to do the extrapolation using some mixture between these reference classes (and others), but in order to get extreme probabilities for extinction you'd need to have an extreme mixture. This is part of the general pattern why you don't usually end up with 99% probabilities for interesting questions without real arguments---you need to not only have a way of estimating that has very high confidence, you need to be very confident in that way of estimating.
  • You could appeal to some similar outside view to say "humanity will undergo changes similar in magnitude to those that have occurred over the last billion years;" I think that's way more plausible (though I still wouldn't believe 99%) but I don't think that it matters for claims about the expected moral value of the future.
  • The doomsday argument can plausibly arrive at very high confidences based on anthropic considerations (if you accept those anthropic principles with very high confidence). I think many long-termists would endorse the conclusion that the vast majority of observers like us do not actually live in a large and colonizable universe---not at 99.999999% but at least at 99%. Personally I would reject the inference that we probably don't live in a large universe because I reject the implicit symmetry principle. At any rate, these lines of argument go in a rather different direction than the rest of your post and I don't feel like it's what you are getting at.
Against GDP as a metric for timelines and takeoff speeds

Scaling down all the amounts of time, here's how that situation sounds to me: US output doubles in 15 years (basically the fastest it ever has), then doubles again in 7 years. The end of the 7 year doubling is the first time that your hypothetical observer would say "OK yeah maybe we are transitioning to a new faster growth mode," and stuff started getting clearly crazy during the 7 year doubling. That scenario wouldn't be surprising to me. If that scenario sounds typical to you then it's not clear there's anything we really disagree about.

Moreover, it seems to contradict your claim that 0.14% growth was already high by historical standards.

0.14%/year growth sustained over 500 years is a doubling. If you did that between 5000BC and 1000AD then that would be 4000x growth. I think we have a lot of uncertainty about how much growth actually occurred but we're pretty sure it's not 4000x (e.g. going from 1 million people to 4 billion people). Standard kind of made-up estimates are more like 50x (e.g. those cited in Roodman's report), half that fast.

There is lots of variance in growth rates, and it would temporarily be above that level given that populations would grow way faster than that when they have enough resources. That makes it harder to tell what's going on but I think you should still be surprised to see such high growth rates sustained for many centuries.

(assuming you discount 1350 as I do as an artefact of recovering from various disasters

This doesn't seem to work, especially if you look at the UK. Just consider a long enough period of time (like 1000AD to 1500AD) to include both the disasters and the recovery. At that point, disasters should if anything decrease growth rates. Yet this period saw historically atypically fast growth.

Against GDP as a metric for timelines and takeoff speeds

Some thoughts on the historical analogy:

If you look at the graph at the 1700 mark, GWP is seemingly on the same trend it had been on since antiquity. The industrial revolution is said to have started in 1760, and GWP growth really started to pick up steam around 1850. But by 1700 most of the Americas, the Philippines and the East Indies were directly ruled by European powers

I think European GDP was already pretty crazy by 1700. There's been a lot of recent arguing about the particular numbers and I am definitely open to just being wrong about this, but so far nothing has changed my basic picture.

After a minute of thinking my best guess for finding the most reliable time series was from the Maddison project. I pulled their dataset from here.

Here's UK population:

  • 1000AD: 2 million
  • 1500AD: 3.9 million (0.14%/year growth)
  • 1700AD: 8.6 million (0.39%)
  • 1820AD: 21.2 million (0.76%)

A 0.14%/year growth rate was already very fast by historical standards, and by 1700 things seemed really crazy.

Here's population in Spain:

  • 1000AD: 4 million
  • 1500AD: 6.8 million (0.11%)
  • 1700AD: 8.8 million (0.13%)
  • 1820AD: 12.2 million (0.28%)

The 1500-1700 acceleration is less marked here but still seems like growth was fast.

Here's the world using the data we've all been using in the past (which I think is much more uncertain):

  • 10000BC: 4 million
  • 3000BC: 14 million (0.02%)
  • 1000BC: 50 million (0.06%)
  • 1000AD: 265 million (0.08%)
  • 1500AD: 425 million (0.09%)
  • 1700AD: 610 million (0.18%)
  • 1820AD: 1 billion (0.41%)

This puts the 0.14%/year growth in the UK in context, and also suggests that things were generally blowing up by 1700AD.

I think that looking at the country-level data is probably better since it's more robust, unless your objection is "GWP isn't what matters because some countries' GDP will be growing much faster."

Load More