Using votes to push towards the score we think it should be at sounds worse than just individually voting according to some thresholds of how good/helpful/whatever a post needs to be? I'm worried about zero sum (so really negative sum because of the effort) attempts to move karma around where different people are pushing in different ways, where it's hard to know how to interpret the results, compared to people straightforwardly voting without regard to others' votes.
At least, if we should be voting to push things towards our best guess I think the karma system should be reformed to something that plays nice with that -- e.g. each individual gives their preferred score, and the displayed karma is the median.
Although early fundraising could be correlational with success rather than causal, if it's an indicator of who can generate support from the electorate.
(I'd be pretty confident there's an effect like this but don't know how strong, and haven't tried to understand if the article you're quoting from tries to correct for it.)
I sort of guess the second thing?; although I never downvoted at least I felt a little defensive and negative about "tone-deaf, indeed chilling" and didn't upvote despite having found your comment useful!
(I've now noticed the discrepancy and upvoted it)
On my impressions: relative to most epistemic communities I think EA is doing pretty well. Relatively to a hypothetical ideal I think we've got a way to go. And I think the thing is good enough to be worth spending perfectionist attention on trying to make excellent.
Some (controversial) reasons I'm surprisingly optimistic about the community:
1) It's already geographically and social-network bubbly and explores various paradigms.
2) The social status gradient is aligned with deference at the lower levels, and differentiation at the higher levels (to some extent). And as long as testimonial evidence/deference flows downwards (where they're likely to improve opinions), and the top-level tries to avoid conforming, there's a status push towards exploration and confidence in independent impressions.
3) As long as deference is... (read more)
Communication channels which allow for lots of information and context to flow back and forth between people. e.g. if I read an article and then go to enact the plan described in the article, that's low-bandwidth. If I sit down with the author for three hours and interrogate them about the reasoning and ask what they think about my ideas for possible plan variations, that's high-bandwidth.
I vibe with the sentiment "particularly uncomfortable with people in the meta space deferring to authority", but I think it's too strong. e.g. I think it's valuable for people to be able to organize big events, and delegate tasks among the event organizers.
Maybe I'm more like "I feel particularly uncomfortable with people in the meta space deferring without high bandwidth, and without explicit understanding that that's what they're doing".
I'm fine with junior ops people at an AI org being not really at all bought into the specific research agenda.
I'm fine with senior technical people not being fully bought in -- in the sense that maybe they think if it were up to them a different agenda would be slightly higher value, or that they'd go about things a slightly different way. I think we should expect that people have slightly different takes, and don't get the luxury of ironing all of those differences out, and that's pretty healthy. (Of course I like them having a go at discussing differences of opinion, but I don't think failure to resolve a difference means that the they need to adopt the party line or go find a different org.)
[Without implying I agree with everything ...]
This comment was awesome, super high density of useful stuff. I wonder if you'd consider making it a top level post?
No, that doesn't work because epistemic deferring is also often about decisions, and in fact one of the key distinctions I want to make is when someone is deferring on a decision how that can be for epistemic or authority reasons, and how those look different.
I agree it's slightly awkward that authorities often delegate, but I think that that's usually delegating tasks; "delegating choices" to me has much less connotation of a high-status person delegating to a low-status person.
Although ... one of the examples of "deferring to authority" in my sense is a ... (read more)
Perhaps "deferring on views" vs "delegating choices" ?
"Deferring to experts" carries the wrong meaning, I think? At least to me that sounds more like epistemic deferring.
An alternative to "deferring to authority" a couple of people have suggested to me is "delegating", which I sort of like (although maybe it's confusing if one of the paradigm examples is delegating to your boss).
So, my immediate reaction is that I can feel that kind of concern, but I think the "see the truth, obey your leaders" is exactly the kind of dynamic I'm worried about! & then I'm trying to help avoid it by helping to disambiguate between epistemic deferring and deferring to authority (because conflating them is where I think a lot of the damage comes from).
So then I'm wondering if I've made some bad branding decisions (e.g. should I have used a different term for what I called "deferring to authority"? It's meant to evoke that someo... (read more)
(fwiw I upvoted this post, because I thought it raised a lot of interesting points that are worth discussing despite disagreeing some bits).
In sum: I think your post sometimes lacks specificity which makes people think you're talking more generally than (I suspect) you are.
Those details matter because ... (read more)
Yeah I briefly alluded to this but your explanation is much more readable (maybe I'm being too terse throughout?).
My take is "this dynamic is worrying, but seems overall less damaging than deferral interfering with belief formation, or than conflation between epistemic deferring and deferring to authority".
I think this is getting downvotes and I'm curious whether this is because:
Thanks! Constructive suggestions about good things to do seem great.
I think Carrick is getting a lot of support from a combination of making crucial issues like pandemic preparedness priorities, and also benefiting from reputation networks here (so people are justifiably confident that he isn't going to be in it for himself or giving out political favours, which is just a super-important dimension). It's certainly plausible that McLeod-Skinner's campaign is a great opportunity to help out with, but my personal impression is that you haven't (yet) made a co... (read more)
I might well be wrong (in which case hopefully someone will correct me), but my understanding is that your second $2,900 would be restricted for use in the general election so couldn't be spent on the remainder of the primary. (Earlier in the campaign there were indirect benefits of total money raised, but it's probably a bit late in the day for those.)
FWIW I've been trying to fly business class for transatlantic flights for a few years for these reasons. I think it's an usually big effect size for me because otherwise long haul flights play badly with my chronic fatigue and can cost me effectively >1 day, but I expect that many people would get a few hours' worth of extra productive time (I take advantage of both the lie-flat bed and the good work environment for writing that doesn't need internet).
I've felt weird about expensing it so mostly just been paying for it myself (I don't have many other bi... (read more)
Thanks, I think this is really valuable.
My opinionated takes for problem solvers:
(1) Over time we'll predictably move in the direction from "need theory builders" to "need problem solvers", so even if you look around now and can't find anything, it might be worth checking back every now and again.
(2) I'd look at ELK now for sure, as one of the best and further-in-this-direction things.
(3) Actually many things have at least some interesting problems to solve as you get deep enough. Like I expect curricula teaching ML to very much not do this, but if you have mastery of ML and are trying to a... (read more)
To me they feel like pre-formal math? Like the discussion of corrigibility gives me a tingly sense of "there's what on the surface looks like an interesting concept here, and now the math-y question is whether one can formulate definitions which capture that and give something worth exploring".
(I definitely identify more with the "theory builder" of Gower's two cultures.)
A handful of ideas (things that tickle my aesthetic) from an ex-topologist:
https://ai-alignment.com/corrigibility-3039e668638 (and other things from https://ai-alignment.com/ )
The second and third strike me as useful ideas and kind of conceptually cool, but not terribly math-y; rather than feeling like these are interesting math problems, the math feels almost like an afterthought. (I've read a little about corrigibility before, and had the same feeling then.) The first is the coolest, but also seems like the least practical -- doing math about weird simulation thought experiments is fun but I don't personally expect it to come to much use.Thank you for sharing all of these! I sincerely appreciate the help collecting data about how existing AI work does or doesn't mesh with my particular sensibilities.
Makes sense; thanks for flagging. I'm tempted to conclude "robustly a bad idea".
Maybe the parameter that I can most imagine someone pushing on to make it look better is that I'm assuming 5% of mineable coal will stay in the ground on default trajectories, and you might think it would be significantly less than that. I don't think this would make it look better than generic clean energy R&D, but it's not impossible (my cost-effectiveness estimate is >1000x below where I'd put the threshold for interventions I'm excited about, so it seems pretty much impossible for it to reach that if my calc is currently skewing optimistic in places).
Thanks, really like this point (which I've kind of applied many times but not had such a clean articulation of).
I think it's important to remember that the log returns model is essentially an ignorance prior. If you understand things about the structure of the problem at hand you can certainly depart from it. e.g. when COVID emerged, nobody had spent any time trying to find and distribute a COVID vaccine. But it will be obvious that going from $1 million to $2 million spent won't give you an appreciable chance of solving the problem (since there's no wa... (read more)
Thanks John. I happen to have done a BOTEC on this; I'll post it here b/c this seems like a canonical place for conversation. It's pretty scrappy, and you shouldn't feel obliged to respond, but I'd be interested to know if you think it's going wrong anywhere (I think my bottom-line is slightly more negative than your "might be a good option for donors who would otherwise have lower impact").
FWIW, your calculation seems still optimistic to me, still, e.g. assuming quite a high elasticity (cost of coal is not such an important part of the cost of producing electricity with coal) and, if I understand your reasoning correctly, a fairly high chance of additionality (by default, coal is in structural decline globally).
Yeah. I don't have more understanding of the specifics than are given on that grant page, and I don't know the theory of impact the grantmakers had in mind, but it looks to me like something that's useful because it feeds into future strategy by "our crowd", rather than because it will have outputs that put the world in a generically better position.
Here's a quick and dirty version of taking the OP grant database and then for ones in the last 9 months categorizing them first by whether they seemed to be motivated by longtermist considerations and for the ones I marked as yes by what phase they seemed to be.
Of 24 grants I thought were unambiguously longtermist, there was 1 I counted as unambiguously Phase 2. There were a couple more I thought might well be Phase 2, and a handful which might be Phase 2 (or have Phase 2 elements) but I was sort of sceptical, as well as three more which were unambiguously... (read more)
Thanks for voicing the frustration!
I regarded the post not really as a point about cause prioritization (I agree longevity research doesn't get much attention, and I think possibly it should get more), but about rhetoric. "Defeating death" seems to be a common motif e.g. in various rationalist/EA fic, or the fable of the dragon tyrant. I just wanted some place which assembled the arguments that make me feel uneasy about that rhetoric. I agree that a lot of my arguments are not really novel intellectual contributions (just "saying the obvious things") and t... (read more)
I certainly feel like it's a very stakesy decision! This is somewhere where a longtermist perspective might be more hesitant to take risks that jeopardize the entire future to save billions alive today.
I also note that your argument applies to past cases too. I'm curious in what year you guess it would first have been good to grant people immortality?
(As mentioned in the opening post, I'm quite confused about what's good here.)
I agree that probably you'd be fine starting today, and it's a much safer bet than starting 1,000 years ago, but is it a safer bet than waiting say another 200 years?
I'd be concerned about dictators inciting violence against precisely the people they most perceive as threats. e.g. I don't know the history of the Cultural Revolution well, but my impression is that something like this happened there.
It's a good point that by default you'd be extending all the great minds too. Abstractly I was tracking this, but I like calling it out explicitly.
& I agree with the trend that we're improving over time. But I worry that if we'd had immortality for the last thousand years maybe we wouldn't have seen the same degree of improvement over time. The concern is if someone had achieved global dictatorship maybe that would have involved repressing all the smart good people, and preventing coordination to overthrow them.
Good questions! I could give answers but my error bars on what's good are enormous.
(I do think my post is mostly not responding to whether longevity research is good, but to what the appropriate attitudes/rhetoric towards death/immortality are.)
Re. term limits on jobs, I think this is a cool idea. But I don't know that I'd expect that to be implemented, which makes want to disambiguate between the questions:
My guesses would be "yes" to A, and a very tentative "no" to B. Of course if there was a now-or-never moment of choosing whether to get immortality, one might still like to have it now, but it seems like maybe we'd ideally wait until society is mature enough that it can handle immortality well before granting it.
I hadn't seen this discussion, thanks! I find the dictator data somewhat reassuring, but only somewhat. Because I care about not the average case dictator, but the tail of dictators having power for a long time. And if say 2% of dictators are such that they'd effectively work out how to have an ironclad grasp of their country that would persist for 1000+ years, I don't really expect our data to be rich enough to be able to pull out that structure.
When thinking about the tail of dictators don't you also have to think of the tail of good people with truly great minds you would be saving from death? (People like John von Neumann, Benjamin Franklin, etc.)
Overall, dictators are in a very tough environment with power struggles and backstabbing, lots of defecting, etc. while great minds tend to cooperate, share resources, and build upon each other.
Obviously, there are a lot more great minds doing good than 'great minds' wishing to be world dictators. And it seems to trend in the right direction. Com... (read more)
Thanks for this; it made me notice that I was analyzing Chris's work more in far mode and Redwood's more in near mode. Maybe you're right about these comparisons. I'd be be interested to understand whether/how you think the adversarial training work could most plausibly be directly applied (or if you just mean "fewer intermediate steps till eventual application", or something else).
Nice! I'm particularly excited by the emphasis on support with choosing degree courses. I think this is important and really underprovided in general.
Have you thought about framing the programme as about helping people who want to have a positive impact with their work rather than about helping young EAs? I'm a little worried about community effects if "joining EA" comes to be perceived as a way to get generic boosts to one's career (and that people who join in circumstances where they didn't really let themselves think about why they might not want to will be worse long-term contributors than if they had space to think clearly about it). But maybe I'm missing some advantages of framing in terms of EAs.
Re. Gripe #3 (/#3.5): I also think AI stuff is super important and that we're mostly not ready for Phase 2 stuff. But I'm also very worried that a lot of work people do on it is kind of missing the point of what ends up mattering ...
So I think that AI alignment etc. would be in a better place if we put more effort into Phase 1.5 stuff. I think that this is supported by having some EA attention on Phase 2 work for things which aren't directly about alignment, but affect the background situation of the world and so are relevant for how well AI goes. Having t... (read more)
Re. Gripe #2: I appreciate I haven't done a perfect job of pinning down the concepts. Rather than try to patch them over now (I think I'll continue to have things that are in some ways flawed even if I add some patches), I'll talk a little more about the motivation for the concepts, in the hope that this can help you to triangulate what I intended:
Jeff's comment (and my reply) covers ~50% of my response here, with the remaining ~50% splitting as ~20% "yeah you're right that I probably have a sampling bias" and ~30% "well we shouldn't be expecting all the Phase 2 stuff to be in this explicitly labelled core, but it's a problem that it's lacking the Phase 1.5 stuff that connects to the Phase 2 stuff happening elsewhere ... this is bad because it means meta work in the explicitly labelled parts is failing to deliver on its potential".
Yeah I did mean "longtermist EA", meaning "stuff that people arrived at thinking was especially high priority after taking a long hard look at how most of the expected impact of our actions is probably far into the future and how we need to wrestle with massive uncertainty about what's good to do as a result of that".
I was here imagining that the motivation for working on Wave wasn't that it seemed like a top Phase 2 priority from that perspective. If actually you start with that perspective and think that ~Wave is one of the best ways to address it, then ... (read more)
I agree with quite a bit of this. I particularly want to highlight the point about combo teams of drivers and analytical people — I think EA doesn't just want more executors, but more executor/analyst teams that work really well together. I think that because of the lack of feedback loops on whether work is really helpful for longterm outcomes we'll often really need excellent analysts embedded at the heart of execute-y teams. So this means that as well as cultivating executors we want to cultivate analyst types who can work well with executors.
That seems archetypically Phase 1 to me? (There's a slight complication about the thing being recruited to not quite being EA)
But I also think most people doing Phase 1 work should stay doing Phase 1 work! I'm making claims about the margin in the portfolio.
I meant to include both (A) and (B) -- I agree that (A) is a bottleneck right now, though I think doing this well might include some reasonable fraction of (B).
One set of examples is in this section of another post I just put up (linked from a footnote in this post), but that's pretty gesturing / not complete.
I think that for lots of this alignment work there's an ambiguity about how much to count the future alignment research community as part of "longtermist EA" which creates ambiguity about whether the research is itself Phase 2. I think that Redwood's work is Phase 1, but it's possible that they'll later produce research artefacts which are Phase 2. Chris Olah's old work on interpretability felt like Phase 2 ... (read more)
I like this. I touched on some similar themes in these old notes on "neutral hours" around different energy levels being more or less productive for different kinds of work, but I didn't really get to the "value of time varies a lot with opportunities", and I think you're right that that's an important part of the puzzle.
I feel like there's just a crazy number of minority views (in the limit a bunch of psychoses held by just one individual), most of which must be wrong. We're more likely to hear about minority views which later turn out to be correct, but it seems very implausible that the base rate of correctness is higher for minority views than majority views.
On the other hand I think there's some distinction to be drawn between "minority view disagrees with strongly held majority view" and "minority view concerns something that majority mostly ignores / doesn't have a view on".
that is a fair point. departures from global majority opinion still seems like a pretty weak 'fire alarm' for being wrong. Taking a position that is eg contrary to most experts on a topic would be a much greater warning sign.
Intuitively I'm pretty interested in the possibility of supporting more formats in service of serious discourse (e.g. having a place to share recordings of conversations that others might benefit from), and pretty uninterested in extra formats for the sake of driving more engagement ... there's a middle ground of "driving engagement with serious discourse" which I'm not sure what to feel about.
If this looks like an issue, one could distinguish speech acts (which are supposed to meet certain standards) from the outputs of various transparency tools (which hopefully meet some standards of accuracy, but might be based on different standards).
The idea is that one statement which is definitely false seems a much more egregious violation of truthfulness than e.g. four statements each only 75% true.
Raising it to a power >1 is a factor correcting for this. The choice of four is a best guess based on thinking through a few examples and how bad things seemed, but I'm sure it's not an optimal choice for the parameter.
The distinction I'm drawing is that "cannot spread it to you" is ambiguous between whether it's shorthand for:
Whereas I think that "can never spread it to you" or "absolutely cannot spread it to you" are harder to interpret as being shortenings of 2.