Interesting take, quick notes:1) I worked on a similar model with Justin Shovelain a few years back. See: https://www.lesswrong.com/posts/BfKQGYJBwdHfik4Kd/fai-research-constraints-and-agi-side-effects2) Rather, one's impact is positive if the ratio of safety and capabilities contributions si/ci is greater than the average of the rest of the world.I haven't quite followed your model, but this doesn't see exactly correct to me. For example, if the mean player is essentially "causing a lot of net-harm", then "just causing a bit of net-harm", clearl... (read more)
Happy to see more discussion on these topics.Much of this is a part of what both some of the EA forecasting community, and what we at [QURI](https://quantifieduncertainty.org/), are working on.I think the full thing is much more work than you think it is. I suggest trying to take one subpart of this problem and doing it very well, instead of taking the entire thing on at once.
Here's my super quick take, if I were evaluating this for funding:Startups are pretty competitive. For me to put money into a business venture, I'd want quite a bit of faith that the team is very strong. This would be pretty high bar. From looking at this, it's not clear to me promising the team is at this point.Generally, the bar for many sorts of projects is fairly high.
A few quick things:- I agree that many grantmakers don't have enough time to give much feedback, and that this leads to suboptimal outcomes.- I think it's pretty difficult for people outside these organizations to help much with what are basically internal processes. People outside have very little context, so I would expect them to have a tough time suggesting many ideas.- In this specific proposal, I think it would be tricky for it to help much. A lot of what I've seen (which isn't all too much) around grant applications is about people sharing the negat... (read more)
This looks really interesting, will take me some time to get through.Very minor thing: Many of the links (the titles) are broken.
I think this is neat. Perhaps-minor note: if you'd do it at scale, I imagine you'd want something more sophisticated than coarse base rates. More like, "For a project that has these parameters, our model estimates that you have a 85% chance of failure."I of course see this as basically a bunch of estimation functions, but you get the idea.
No worries! I assumed as such.
Minor note about the name: "Never Again" is a slogan often associated with the Holocaust. I think that people using it for COVID might be taken as appropriation or similar. I might suggest a different name. https://en.wikipedia.org/wiki/Never_again
One quick thought; often, when things are very grim, you're pretty okay taking chances.
Imagine we need 500 units of AI progress in order to save the world. In expectation, we'd expect 100. Increasing our amount to 200 doesn't help us, all that matters is if we can get over 500. In this case, we might want a lot of bifurcation. We'd much prefer a 1% chance of 501 units, than a 100% chance of 409 units, for example.
In this case, lots of randomness/bifurcation will increase total expected value (which is correlated with our chances of getting over 500 units, ... (read more)
It seems to me like quantum randomness can be a source of creating legitimate divergence of outcomes. Lets call this "bifurcation". I could imagine some utility functions for which increasing the bifurcation of outcomes is beneficial. I have a harder time imagining situations where it's negative.
I'd expect that interventions that cause more quantum bifurcation generally have other costs. Like, if I add some randomness to a decision, the decision quality is likely to decrease a bit on average.So there's a question of the trade-offs of a decrease in th... (read more)
My impression is that you're arguing that quantum randomness creates very large differences between branches. However, couldn't it still be the case that even more differences would be preferable? I'm not sure how much that first argument would impact the expected value of trying to create even more divergences.
So the incentives are not pointing in the right direction. Capable forecasters can earn significantly more by predicting societally-useless sports stuff, or simply by arbitraging between the big European sports-houses and crypto markets. Meanwhile, the people who remain forecasting socially useful stuff on Metaculus, like whether Russia will invade the Ukraine or whether there will be any new nuclear explosions in wartime, do so to a large extent out of the goodness of their heart.I think that the clear solution to this is to either increase the overall wi
So the incentives are not pointing in the right direction. Capable forecasters can earn significantly more by predicting societally-useless sports stuff, or simply by arbitraging between the big European sports-houses and crypto markets. Meanwhile, the people who remain forecasting socially useful stuff on Metaculus, like whether Russia will invade the Ukraine or whether there will be any new nuclear explosions in wartime, do so to a large extent out of the goodness of their heart.
I think that the clear solution to this is to either increase the overall wi
This is an interesting idea, thanks for raising it!I think intuitively, it worries me. As someone around hiring in these sorts of areas, I'm fairly nervous around the liabilities that come from hiring, and this seems like it could increase these. (Legal, and just upsetting people).I'm imagining:
That makes more sense, thanks!
Quick thoughts of possible improvements to the format:1) Make both the start time and the end time clear2) Include a link to the person's website/linkedin. Right now I just search each person and choose whatever is on top of Google, anyway. Much of the grants really depend on the specific person, so linking to more information about the person would be valuable. (I realize this might be a bit annoying in terms of getting their buy-in/information)
Just to clarify:That last report was for May 2021.So does this report mainly cover grants for June and July, only?
Confusingly, the report called "May 2021" was for grants we made through March and early April of 2021, so this report includes most of April, May, June, and July.I think we're going to standardize now so that reports refer to the months they cover, rather than the month they're released.
This post is relevant: https://www.lesswrong.com/posts/vCQpJLNFpDdHyikFy/are-the-social-sciences-challenging-because-of-fundamental
The health interventions seem very different to me than the productivity interventions.The health interventions have issues with long time-scales, which productivity interventions don't have as much.However, productivity interventions have major challenges with generality. When I've looked into studies around productivity interventions, often they're done in highly constrained environments, or environments very different from mine, and I have very little clue what to really make of them. If the results are highly promising, I'm particularly skeptical, so i... (read more)
I think the obvious answer is that doing controlled trials in these areas is a whole lot of work/expense for the benefit.Some things like health effects can take a long time to play out; maybe 10-50 years. And I wouldn't expect the difference to be particularly amazing. (I'd be surprised if the average person could increase their productivity by more than ~20% with any of those)On "challenge trials"; I imagine the big question is how difficult it would be to convince people to accept a very different lifestyle for a long time. I'm not sure if it's called "challenge trial" in this case.
That sounds like much of it. To be clear, it's not that the list is obvious, but more that it seems fairly obvious that a similar list was possible. It seemed pretty clear to me a few years ago that there must be some reasonable lists of non-info-hazard countermeasures that we could work on, for general-purpose bio safety. I didn't have these particular measures in mind, but figured that roughly similar ones would be viable.
Another part of my view is,"Could we have hired a few people to work full-time coming up with a list about this good, a few year... (read more)
Really happy to see this, this looks great. This is outside the scope of this document, but I'm a bit curious how useful it would have been to have such a list 3-5 years ago, and why it took so long. Previously I heard something like, "biosecurity is filled with info-hazards, so we can't have many people in it yet."Anyway, it makes a lot of sense to me that we have pretty safe intervention options after all, and I'm happy to see lists being created and acted upon.
The authors will have a more-informed answer, but my understanding is that part of the answer is "some 'disentanglement' work needed to be done w.r.t. biosecurity for x-risk reduction (as opposed to biosecurity for lower-stakes scenarios)."
I mention this so that I can bemoan the fact that I think we don't have a similar list of large-scale, clearly-net-positive projects for the purpose of AI x-risk reduction, in part because (I think) the AI situation is more confusing and requires more and harder disentanglement work (some notes on this here and here). Th... (read more)
I have only been involved in biosecurity for 1.5 years, but the focus on purely defensive projects (sterilization, refuges, some sequencing tech) feels relatively recent. It's a lot less risky to openly talk about those than about technologies like antivirals or vaccines.I'm happy to see this shift, as concrete lists like this will likely motivate more people to enter the space.
Yay, thanks so much!
I don't really use the word myself (at least, I don't remember using it), but I sometimes do say things like "intense utilitarian" or "intense worker."I'd vote against "Drank the kool-aid EAs.” It's a super dark metaphor; of an altruistic sect that turned into a cult and committed mass suicide. I get that it's joking, but it feels like a bit much for me. https://en.wikipedia.org/wiki/Drinking_the_Kool-Aid
The phrase originates from events in Jonestown, Guyana, on November 18, 1978, in which over 900 members of the Peoples Temple movem
On the whole I liked this a lot, and I broadly agree. Around "academics being too optimistic": I've seen similar a few times before and am pretty tired of it at this point. I'm happy that interesting ideas are brought forward, but I think the bias is pretty harmful. In fairness though, this is really a community issue; if our community epistemics were better, than the overconfidence of academic takes wouldn't have lead to much overconfidence of community beliefs.Some thoughts:1. I agree that the implementation of "general purpose many-emplo... (read more)
I think market sentiment is a bit complicated. Very few investors are talking about AGI, but organizations like OpenAI still seem to think that talking about AGI is good marketing for them (for talent, and I'm sure for money, later on). I think most of the Anthropic investment was from people close to effective altruism: Jaan Tallinn, Dustin Moskovitz, and Center for Emerging Risk Research, for example. https://www.anthropic.com/news/announcement
On why those people left OpenAI, I'm not at all an expert here. I think it's common for different tea... (read more)
That all sounds pretty good to me. I like the idea of a wide variety of means of support; both to try out more things (it's hard to tell what would work in advance), and because it's probably a better solution long-term.
Will do. No one comes to mind now, but if someone does, I'll let you know.(Also, I'm sure others reading this with ideas should send them to Bob)
Good point about focusing on money; this post was originally written differently, then I tried making it more broad, but I think it wound up being more disjointed than I would have liked.
First, I’d also be very curious about interventions other than money.
Second though, I think that “money combined with services” might be the most straightforward strategy for most of the benefits except for friends.
“Pretty strong services” to help set up people with mental and physical health support could exist, along with insurance setups. I think that setting up new ser... (read more)
I think this is a serious question.
One big question is is this would be viewed more as a "community membership" thing or as a "directly impactful" intervention. I could imagine both being pretty different from one another.I think personally I'm more excited by the second, because it seems more scalable. The way I would view the "utilitarian intervention" version would be pretty intense, and much unlike almost all social programs, but it would be effective.1. "Fairness" is a tricky word. The main thing that matters is who's expected to produce value.&n... (read more)
Yep. Sorry, I didn't mean to make it seem like it was. Changed.
(Just noting general agreement with this)
I agree that your proposal gets around most (maybe all?) of the issues I mentioned. However, your proposal focuses on earning-to-givers who have already given a fair bit, this seems to be tackling a minority of the problem (maybe 20%?). Maybe this is a good place to begin. I feel like I haven't met many people in this specific camp, but maybe there are more out there.
Do you agree with this?
I'm happy to see it on a small scale. That said, the existing discussion/debate doesn't seem like all too much to me. I also feel like there could be some ea... (read more)
Agreed that higher salaries could help (and are already helping). Another nice benefit is that they can also be useful for the broader community; more senior people will have more money to help out more junior people, here and there.I imagine if there were an insurance product, it would be subsidized a fair amount. My hope would be that we could have more trust than would exist for a regular insurance agency, but I'm not sure how big of a deal this would make.
Yea; this was done with a search was for "AGI". There's no great semantic search yet, but I could see that as a thing in the future.I added a quick comment in this section about it.
+1That said, I think I might even more prefer some sort of emoji system, where there were emojis to represent each of the 4 dimensions, but also the option to have more emojis.
Epistemic status: I feel positive about this, but note I'm kinda biased (I know a few of the people involved, work directly with Nuno, who was funded)ACX Grants just announced.~$1.5 Million, from a few donors that included Vitalik.
Relevant:I just came across this LessWrong post about a digital tutor that seemed really impressive. It would count there.https://www.lesswrong.com/posts/vbWBJGWyWyKyoxLBe/darpa-digital-tutor-four-months-to-total-technical-expertise
I might be able to provide a bit of context:I think the devil is really in the details here. I think there are some reasonable versions of this. The big question is why and how you're criticizing people, and what that reveals about your beliefs (and what those beliefs are).As an extreme example, imagine if a trusted researcher came out publicly, saying,"EA is a danger to humanity because it's stopping us from getting to AGI very quickly, and we need to raise as much public pressure against EA as possible, as quickly as possible. We need to shut EA dow... (read more)
This is all reasonable but none of your comment addresses the part where I'm confused. I'm confused about someone saying something that's either literally the following sentence, or identical in meaning to: "Please don't criticize central figures in EA because it may lead to an inability to secure EA funding."
If I were a funder, and I were funding researchers, I'd be hesitant to fund researchers who both believed that and was taking intense action accordingly. Like, they might be directly fighting against my interests.
That part of the example ma... (read more)
Interesting, thanks for sharing! I imagine many others here (myself included) will be skeptical of the parts:1. Narrow AI will be just as good, particularly for similar development costs. (To me it seems dramatically more work-intense to make enough narrow AIs)2. The idea that really powerful narrow AIs won't speed up AGIs and similar. 3. Your timelines (very soon) might be more soon than others here, though it's not clear exactly what "possible" means. (I'm sure some here would put some probability mass 5-10yrs out, just a different amount) ... (read more)
This sounds a lot like #3 and #1 to me. It seems like you might have slightly unique intuitions on the parameters (chance of success, our ability to do things about it), but the rough shape seems similar.
Fair point! I think I'll instead just encourage people to read the comments. Ideally, more elements will be introduced over time, and I don't want to have to keep on updating the list (and the title).
Interesting, I wasn't at all thinking about the orthogonality thesis or moral realism when writing that. I was thinking a bit about people who:1) Start out wanting to do lots of tech or AI work.2) Find out about AGI and AGI risks.3) Conclude on some worldview where doing lots of tech or AI work is the best thing for AGI success.
An, that’s really good to know… and kind of depressing.
Thanks so much.
Good idea. I think it's difficult to encourage people to write a huge amount of data on a website like that. Maybe you could scrape forums or something to get information.I imagine that some specific sorts of decisions will be dramatically more tractable to work on than others.
You seem to have a small formatting mistake in the link, this should work though.https://www.skynettoday.com/editorials/dont-worry-agi/
Some quick thoughts:
1. Kudos for donating that much, even with a direct work position.
2. I've been helping out Patrick a bit, and wound up deciding between Longview and EA Funds. Both seem pretty strong and like they could absorb more money. If you don't want to spend too much time on deciding, these seem pretty safe. (Note that longview is longtermist)
3. EA is vetting constrained, and I'd guess you would be better than many at vetting (particularly those who aren't currently funders). I'd be curious what you'd come up with if you were to spend time on thi... (read more)
Good points. I think I agree that being able to offer grants in between $1k-$5k seems pretty useful. If they get to be a pain, I imagine there will be ways to lessen the marginal costs.
Sorry if my post made it seem that way, but I don't feel like I've been thinking of it that way. In fact, it's sort of worse if it's not a single actor; many different departments could have done something about this, but none of them seemed to take public action.
I'm not sure how to understand your second sentence exactly. It seems pretty different from your first sentence, from what I can tell?
A multi-actor system is constrained in ways that a group of single actors are not. Individual agencies can't do their own thing publicly, and you can't see what they are doing privately.For the agencies that do pay attention, they can't publicly respond - and the lack of public monitoring and response by government agencies which can slap new regulations on individual companies or individuals is what separates a liberal state from a dictatorship. If US DOD notices something, they really, really aren't allowed to respond publicly, especially in ways that wo... (read more)
I don't know if there's a high-leverage point where a few EAs or even the entire EA community can come in and bring a lot of change.
From my perspective, most decision automation is highly neglected for some reason. I don't know why, but things seem to be moving really slowly right now, especially for the big picture sorts of decisions effective altruists care about. I don't know of any startups trying to help people make career decisions using probability distributions / expected values, for example. (Or most of the other questions I listed in this documen... (read more)