All of OscarD's Comments + Replies

Great post, and an interesting counterfactual history!

Hooray for moral trade.

Evolutionary debunking arguments feel relevant re the causal history of our beliefes.

One thing I have heard is that having long-ish application stages provides value by getting more people to think about relevant topics (I have heard this from at least two orgs I think). E.g. having several hundred people spend an hour writing a paragraphs about an AI safety topic might be good by virtue of generally having more people think more about this being good. I haven't seen a write-up weighing up the pros and cons of this though. I agree this can be bad for applicants.

NIce post!

We might then expect a lot of powerful attempts to change prevailing ‘human’ values, prior to the level of AI capabilities where we might have worried a lot about AI taking over the world. If we care about our values, this could be very bad. 

This seems like a key point to me, that it is hard to get good evidence on. The red stripes are rather benign, so we are in luck in a world like that. But if the AI values something in a more totalising way (not just satisficing with a lot of x's and red stripes being enough, but striving to make all hum... (read more)

Hmm true, I think I agree that this means the dynamics I describe matter less in expectation (because the positional goods-oriented people will be quite marginal in terms of using the resources of the universe).

Good point re aesthetics perhaps mattering more, and about people dis-valuing inequality and therefore not wanting to create a lot of moderately good lives lest they feel bad about having amazing lives and controlling vast amounts of resources.

Re "But I don't think ..." in your first paragraph, I am not sure what if anything we actually disagree about. I think what you are saying is that there are plenty of resources in our galaxy, and far more beyond, for all present people to have fairly arbitrarily large levels of wealth. I agree, and I am also saying that people may want to keep it roughly that way, rather than creating heaps of people and crowding up the universe.

Oh nice, I hadn't seen that one, thanks!

Nice, good idea and well implemented!

In terms of wastewater being good for getting samples from lots of people at once and not needing ethics clearance, but being worse for respiratory pathogens, how feasible is airborne environmental DNA sampling? I have never looked into it, I just remember hearing someone give a talk about their work on this, I think related to this paper: https://www.sciencedirect.com/science/article/pii/S096098222101650X

I assume it is just hard to get the quantity of nucleic acids we would want from the air.

Flagging this for @Conrad K... (read more)

Thanks for writing this up! Have you spoken to Christian Ruhl or anyone else at Founder's Pledge about this work? I think FP would be interested in and benefit from this.

I downvoted because there are lots of questions lumped in together without enough motivation and cohesion for my liking, and compared to e.g. the moral weights project the engagement with these subtle issues feels more flippant than serious.

Makes sense, sounds good!

Nice post! Re the competitive pressures, this seems especially problematic in long-timelines worlds where TAI is really hard to build, as (toy model) if company A spends all its cognitive assets on capabilities (including generating profit to fund this research), while company B spends half its cognitive assets at any given time on safety work with no capabilities overflows, then if there is a long time over which this exponential growth continues, company A will likely reach the lead even if it starts well behind. Whereas if there is a relatively smaller ... (read more)

Exciting! Why the relocation from Switzerland to the UK? The fact that there are more EA/X-risk projects already in London seems like both a pro (more networking and community opportunities, better access to mentors) and a con (less differentiation with other projects like ERA and MATS, less neglected than mainland Europe fellowships).

Feel free to not reply if you deliberately don't want to make this reasoning public.

My guess now of where we most disagree is regarding the value of a world where AIs disempower humanity and go onto have a vast technologically super-advanced, rapidly expanding civilisation. I think this would quite likely be ~0 value since we don't understand consciousness at all really, and my guess is that AIs aren't yet conscious and if we relatively quickly get to TAI in the current paradigm they probably still won't be moral patients. As a sentientist I don't really care whether there is a huge future if humans (or something sufficiently related to h... (read more)

To red-team a strawman of your (simulated) argument: what about the Pascallian and fanatical implications across evidentially cooperating large worlds? I think we need some Bayesian, anthropic reasoning, lots of squiggle notebooks, and perhaps a cross-cause cost-effectiveness model to get to the bottom of this!

Thanks, interesting idea, I think I mostly disagree and would like to see AI progress specifically slowed/halted while continuing to have advances in space exploration, biology, nuclear power, etc and that if we later get safe TAI we won't have become too anti-technology/anti-growth to expand a lot. But I hadn't thought about this before and there probably is something to this, I just think it is most likely swamped by the risks from AI. It is a good reason to be careful in pause AI type pitches to be narrowly focused on frontier AI models rather than tech... (read more)

That makes sense, yes perhaps there are some fanaticism worries re my make-the-future large approach even more so than x-risk work, and maybe I am less resistant to fanaticism-flavoured conclusions than you. That said I think not all work like this need be fanatical - e.g. improving international cooperation and treaties for space exploration could be good in more frames (and bad is some frames you brought up, granted).

I don't know lots about it, but I wonder if you prefer more of a satisficing decision theory where we want to focus on getting a decent out... (read more)

Thanks for this really thoughtful engagement! I expected this would not be a take particularly to your liking, but your pushback is stronger than I thought, this is useful to hear. Perhaps I failed to realise how controversial and provocative these ideas would be after playing with them myself and with a few relatively similar people. Onto the substance:

• That makes sense to me that the analogy is a bit weak, I think I mostly agree. I think the strongest part of the analogy to me is less the NIMBYs themselves and more who is politically empowered (a smaller

I was disappointed GiveDirectly wasn't mentioned given that seems to be more what he would favour. The closing anecdote about the surfer-philosopher donating money to Bali seems like a proto-GiveDirectly approach but presumably a lot less efficient without the infrastructure to do it at scale.

Thanks for sharing, it sucks that you went through this (and sucks that the moths went through this :( ). As uncomfortable as thinking about these topics is, I am glad to be part of a community of people who take ethics seriously and try to act with compassion and consideration. Let's hope market forces take effect and enough people inquiring about low-suffering ways to kill insects creates a market for companies to offer this :)

Nice!

I think this makes good sense as a toy theoretical model, and updates me some way towards these conclusions, but not very far because this sort of armchair theorising (while valuable and fun) is hard to get accurate for something as messy and empirical as this, as you note. So I think if someone were to investigate this further the key steps would be to:

• look at empirical literature, or conduct primary research, on pleasure/pain symmetry and whether this holds (maybe this would be intractable though)
• do some more involved population dynamics modelling,

Thanks for writing this! I agree that bioanchors is still worth engaging with and revisiting given how important it has been and is.

I like the overall approach of trying to quantify how much different criticism would update the 1e41 estimate. I don't feel well-placed to comment on the thermodynamic approach part, but if it works roughly as you outline this seems like an important robustness check for the evolution anchor.

I left a bunch of more minor comments in the report.

I think this is a good framing! And I think I am happy to bite this bullet and say that for the purposes of deciding what to do it matters relatively little whether my action being effective relies on systems of humans acting predictably (like polio vaccine deliverers getting paid to do their job) or natural forces (atmospheric physics for a climate geoengineering intervention). Whereas regarding what is a virtuous attitude to have, yes probably it is good to foreground the many (sometimes small) contributions of other humans that help our actions have their desired impacts.

Yes I think that makes sense. I think for me the area where I am most sympathetic to your collective rationality approach is voting, where as you noted elsewhere the 80K narrow consequentialist approach is pretty convoluted. Conversely, the Categorical Imperative, universalisability perspective is very clear that voting is good, and thinking in terms of larger groups and being part of something is perhaps helpful here. So yes while I still generally prefer the counterfactual perspective, I am probably not fully settled there.

I suppose in theory being part ... (read more)

Yes, I think this issue of how many people you need to get on board with the vision/goals to make some change happen is key (and perhaps a crux). I agree the number of people needed to implement a change might be huge (all the farm workers making changes for various animal welfare things) but think we probably don't need to get all of them to care a lot more about nonhumans to get the job done. So in my view often a small-ish set of people advocate for/research/fund/plan some big change, and then lots of people implement it because they are told to/paid to.

Makes sense, I think I don't know enough to continue this line of reasoning that sensibly!

On 2, I like this point about the distribution being shaped by the choices of others, I think it is quite true that if more people cared about impact it would be a lot harder to counterfatually achieve very high impact actions (because there would be so much 'competition' with other impact seekers). Reminiscent of how financial markets are pretty efficient because so many people are seeking to make money trading - I think if a similar number of people were looking to succeed in the 'impact market' there wouldn't be these super cost-effective low-hanging fr... (read more)

Cool, great you had a go at this! I have not had a look at your new code yet (and am not sure I will) but if I do and I have further comments I will let you know :)

Good to hear! Yes I imagine having 50+ comments, many of them questioning/pushing-back, could be a bit overwhelming, from my perspective and I am guessing for others as well it is fine and reasonable if you choose not to engage now/ever. Putting this essay out into the world has already been a useful contribution to the discourse I think :)

Finally, I really hope you do choose to stay at least somewhat involved in ~EA things, as you say having the added intellectual diversity is valuable I think. You are probably the sometimes-critic of EA conventions/dogmas whose views I am most moved by.

Re the polio vaccine, I don't know much about it, but I think the inventors probably do deserve a lot of credit! Yes, lots and lots of people were needed to manufacture and distribute many vaccine doses, but I think the counterfactual is illustrative: the workers driving the trucks and going door to door and so forth seem very replaceable to me and it is hard to imagine a great vaccine being invented, but then not being rolled our because no-one is willing to take a job as a truck driver distributing the doses. Whereas if the inventors didn't invent it, ma... (read more)

I think elitism and inequality are real worries - I think it is lamentable but probably true that some people's lives will have far greater instrumental effects on the world than others. (But this doesn't change their intrinsic worth as an experiencer of emotions and haver of human connections.)

So I agree that there is a danger of thinking too much of oneself as some sort of ubermensch do-gooder, but the question of to what extent impact varies by person or action is separate.

Footnote 5 predicted perfectly the sort of thing I was going to say in response. You probably know more economics than I do, but I feel like there are some models of how markets work that quite successfully predict macro behaviour of systems without knowing all the local individual factors? E.g. re your suggestion that nurses are a large fraction of the 'highest impact' career paths, I think we could run some decent calculations about the elasticity of the nursing labour market to find how many more nurses there will overall be if I decide to be a nurse in... (read more)

I do not agree that there are vast differences in value among those actions and strategies that have crossed the bar of having a significant positive impact on the world

(emphasis added)

Perhaps this is a strawman of your position, but it sounds a bit like you want to split actions into basically three buckets: negative, approximately neutral, and significantly positive. This seems unhelpful to me, for several reasons:

• I think it is uncontroversial that at least on the negative side of the scale some actions are vastly worse than others, e.g. a mass murder or

An overarching thought, not responding to any particular quote from you: I think lots of people in the world (the vast majority in fact!) don't really think about impartial altrusitic impact, let alone maximising it. If this is right, I think it would be a priori not so surprising if there are lots of high-impact opportunities left on the table by most people, waiting for ~EAs to action. Perhaps the clearest case here is something like shrimp or insect welfare. By some lights at least this is very high impact, but it makes sense it wasn't already being worked on because primarily only people with an ~EA mindset would be interested in it.

Wow great essay Sarah, very thought-provoking and relevant I thought.

I have lots of things to say, I will split them into separate comments in case you want to reply to specific parts (but feel free to reply to none of it, especially given I see you have a dialogue coming soon). Or we can just discuss it all on our next call :) But I thought I would write them down while I remember.

Thanks for writing this! Indeed counterfactuals are hard. I have also joined a large EA org (Rethink Priorities) and so far agree it is useful. I think a possible failure mode for me is that I am a bit risk-averse, and also just really like working with EAs, so I'm guessing if in X months/years time I have the option to go off and start/do something by myself or with a small group I might be reluctant to leave a nice, comfortable, convenient, EA org like RP. But I agree there are lots of advantages to working at an established org, at least for a while at the start of my career.

Woohoo, wonderful news and thanks for your efforts!

CHERI is also planning to run this year I believe, for anyone looking to do non-AI projects (I am not involved with CHERI).

Nice! I realised that I can't think of the last time I received low-quality criticism (but can think of a moderate amount of fairly high-quality criticism) so I am probably quite lucky in that regard, as my work/writing thus far has either been privately shared or public but not very provocative. (Of course the flipside is having more people engage with one's writing is one way to increase impact.)

I hadn't heard the "idea inoculation" term before - that does seem like a useful framing. I wonder if that is part of the explanation for some of the AI safety/x... (read more)

Good on you for being courageous and scout-minded enough to shut this down (and to start it in the first place)! I hope you find great projects to move onto.

I quite like the summary bot, and think it would often be useful (particularly for posts without author-written summaries) to read the summary first before deciding to read the whole post. Of course, it is easy to scroll all the way down, read the summary, and then decide whether to read the post. But humans are lazy and to make the user experience as frictionless as possible, how about the AI-written summary goes at the top, above the post? Not everyone would like this, so I think there should be an option for each user whether they want the summary at th... (read more)

Nice! I was surprised that more present-day harms were not more front of mind for respondents (e.g. job losses, AI pornography, and racial and gender bias were far below preventing catastrophic outcomes). Interesting.

What updates are you thinking of? Gemini 1.5?

Thanks for writing up a version for the forum, and congrats on finishing your thesis!

I thought this was useful and clearly written. I particularly liked the discussion of the tension between BWC Articles IV and X, which I hadn't thought about. And very interesting re your detailed digging into IGSC companies and that many of them don't take it very seriously. Shows gov regulation is more important, perhaps. That would be wild if the companies would actually be contractually obliged to not deny dangerous orders in some case!! I know next to nothing about la... (read more)

Thanks for sharing, yes motivational benefits do seem important too!

Nice, I didn't know about some of these, good to take stock after an eventful year! I am so used to GPT-4 and integrating it into my work and life that it is weird to think it has been around such a short length of time ...

Thanks good points, I don't think we disagree directionally, perhaps just on how important some of these effects are. It feels like a very difficult epistemic problem to attribute how much the relative absence of bioweapons use is attributable to the BWC - I know roughly nothing about exploding bullets and the like, but maybe they are just more useful than bioweapons for most belligerants? And therefore are used more irrespective of how strong the relevant treaties are. But yes, agree that these aspects still provide some value :)

Thanks, useful thoughts, I think I roughly agree with you and will change this. I suppose the tradeoff I was facing with the title (not that I spent any time weighing up different options consciously) is between brevity, accurateness, and interestingness. I think the more complete title would be something like 'Updating weakly against the Biological Weapons Convention being as important to work on as I thought'. I think I will change the title to 'Reflections on the BWC' so that people who only see the title don't get a negative vibe (I agree we want peopl... (read more)

Thanks for writing this, I thought it was moving and beautifully written. I think the world would be a lot better if more people showed this sort of radical empathy.