All of aogara's Comments + Replies

Some unfun lessons I learned as a junior grantmaker

“ People are often grateful to you for granting them money. This is a mistake.”

How would you recommend people react when they receive a grant? Saying thank you simply seems polite and standard etiquette, but I agree that it misportrays the motives of the grantmaker and invites concerns of patronage and favoritism.

[half baked idea] It seems reasonable to thank someone for the time they spent evaluating a grant, especially if you also do it when the grant is rejected (though this may be harder). I think it is reasonable to thank people for doing their job even (maybe especially?) when you are not the primary beneficiary of that job, and that their reason for doing it is not thanks.
We Ran an AI Timelines Retreat

Really cool! I was hoping to attend but had to be home for a family event. Would be super interested to see any participants summarize their thoughts on AI timelines, or a poll of the group's opinions. 

DeepMind’s generalist AI, Gato: A non-technical explainer

Sounds like Decision Transformers (DTs) could quickly become powerful decision-making agents. Some questions about them for anybody who's interested: 

DT Progress and Predictions

Outside Gato, where have decision transformers been deployed? Gwern shows several good reasons to expect that performance could quickly scale up (self-training, meta-learning, mixture of experts, etc.). Do you expect the advantages of DTs to improve state of the art performance on key RL benchmark tasks, or are the long-term implications of DTs more difficult to measure? Focusi... (read more)

LW4EA: Some cruxes on impactful alternatives to AI policy work

Cool arguments on the impact of policy work for AI safety. I find myself agreeing with Richard Ngo’s support of AI policy given the scale of government influence and the uncertain nature of AI risk. Here’s a few quotes from the piece.

How AI could be influenced by policy experts:

in a few decades (assuming long timelines and slow takeoff) AIs that are less generally intelligent that humans will be causing political and economic shockwaves, whether that's via mass unemployment, enabling large-scale security breaches, designing more destructive weapons, psycho

... (read more)
DeepMind’s generalist AI, Gato: A non-technical explainer

This is a terrific distillation, thanks for sharing! I really like the final three sections with implications for short-term, long-term, and policy risks. 

For example, in 2019 the U.S. Food and Drug Administration issued a proposed regulatory framework for AI/ML-based software used in health care settings. Less than a week ago, the U.S. Justice Department and the Equal Employment Opportunity Commission released guidance and technical assistance documents around avoiding disability discrimination when using AI for hiring decisions.

These are some great ... (read more)

Sort forum posts by: Occlumency (Old & Upvoted)

I agree, upvotes do seem a bit inflated. It creates an imbalance between new and old users that continually grows as existing users rack up more upvotes over time. This can be good for preserving culture and norms, but as time goes on, the difference between new and old users only grows. Some recalibration could help make the site more welcoming to new users.

In general, I think it would be nice if each upvote counted for roughly 1 karma. Will MacAskill’s most recent post received over 500 karma from only 250 voters, which might exaggerate the reach of the... (read more)

The users with the highest karma [] come from a range of different years, and the two highest joined in 2017 and 2019. I don't think it's too much of a problem.
A hypothesis for why some people mistake EA for a cult

Hey Aman, thanks for the post. It does seem a bit outdated that the top picture for altruism  is a French painting from hundreds of years ago. EA should hope to change the cultural understanding of doing good from something that's primarily religious or spiritual, to something that can be much more scientific and well-informed. 

I do think some of the accusations of EA being a cult might go a bit deeper. There aren't many other college clubs that would ask you to donate 10% of your income or determine your career plans based on their principles. O... (read more)

Thanks for the comment! I agree with your points--there are definitely elements of EA, whether they're core to EA or just cultural norms within the community, that bear stronger resemblances to cult characteristics. My main point in this post was to explore why someone who hasn't interacted with EA before (and might not be aware of most of the things you mentioned) might still get a cult impression. I didn't mean to claim that the Google search results for "altruism" are the most common reason why people come away with a cult impression. Rather, I think that they might explain a few perplexing cases of cult impressions that occur before people become more familiar with EA. I should have made this distinction clearer, thanks for pointing it out :)
EA will likely get more attention soon

This is a great point. As one example of growing mainstream coverage, here’s a POLITICO Magazine piece on Carrick Flynn’s Congressional campaign. It gives a detailed explanation of effective altruism and longtermism, and seems like a great introduction to the movement for somebody new. The author sounds like he might have collaborated with CEA, but if not, maybe someone should reach out?

What are your recommendations for technical AI alignment podcasts?

AXRP by Daniel Filan from CHAI is great, and The Gradient is another good one with both AI safety and general interest AI topics.

The best $5,800 I’ve ever donated (to pandemic prevention).

Coverage of this post from The Hill on April 24th:  

Many of Flynn’s donors are involved in an online forum called Effective Altruism, a group that analyzes how best to spend money on philanthropic efforts. Their conclusion, according to some of the posts backing Flynn, has been that spending a few million on a congressional race could result in billions in spending on pandemic preparedness by the federal government.

Flynn is “the first person to ever run for US congress on a platform of preventing future pandemics,” wrote one user, Andrew Snyder-Beatti

... (read more)
Why Helping the Flynn Campaign is especially useful right now

Donated because of this post. Thanks for sharing and good luck to Carrick.

Axby's Shortform

Hey, this is a great question with good context for potential answers too. If you don’t get any substantive responses here, I’d suggest posting as a question on the frontpage — the shortforms really don’t get much visibility.

Information security considerations for AI and the long term future

Great overview of an important field for AI safety, thanks for sharing. A few questions if you have the time:

First, what secrets would be worth keeping? Most AI research today is open source, with methods described in detailed papers and code released on GitHub. That which is not released is often quickly reverse-engineered: OpenAI’s GPT-3 and DALLE-2 systems, for example, both have performant open-source implementations. On the other hand, many government and military applications seemingly must be confidential. 

What kinds of AI research is kept secr... (read more)

3Jeffrey Ladish19d
I agree that a lot of the research today by leading labs is being published. I think the norms are slowly changing, at least for some labs. Deciding not to (initially) release the model weights of GPT-2 was a big change in norms iirc, and I think the trend towards being cautious with large language models has continued. I expect that as these systems get more powerful, and the ways they can be misused gets more obvious, norms will naturally shift towards less open publishing. That being said, I'm not super happy with where we're at now, and I think a lot of labs are being pretty irresponsible with their publishing. The dual-use question is a good one, I think. Offensive security knowledge is pretty dual-use, yes. Pen testers can use their knowledge to illegally hack if they want to. But the incentives in the US are pretty good regarding legal vs. illegal hacking, less so in other countries. I'm not super worried about people learning hacking skills to protect AGI systems only to use those skills to cause harm -- mostly because the offensive security area is already very big / well resourced. In terms of using AI systems to create hacking tools, that's an area where I think dual-use concerns can definitely come into play, and people should be thoughtful & careful there. I liked your shortform post. I'd be happy to see people apply infosec skills towards securing nuclear weapons (and in the biodefense area as well). I'm not very convinced this would mitigate risk from superintelligent AI, since nuclear weapons would greatly damage infrastructure without killing everyone [] , and thus not be very helpful to eliminating humans imo. You'd still need some kind of manufacturing capability in order to create more compute, and if you have the robotics capability to do this then wiping out humans probably doesn't take nukes - you could do it with drones or bioweapons
There are currently more than 100 open EA-aligned tech jobs

Thanks for sharing. It seems like the most informed people in AI Safety have strongly changed their views on the impact of OpenAI and Deepmind compared to only a few years ago. Most notably, I was surprised to see ~all of the OpenAI safety team leave for Anthropic . This shift and the reasoning behind it have been fairly opaque to me, although I try to keep up to date. Clearly there are risks with publicly criticizing these important organizations, but I'd be really interested to hear more about this update from anybody who understands it.

Big EA and Its Infinite Money Printer

Very important perspective from someone on the front lines of recruiting new EAs. Thanks for sharing!

How effective is sending a pre-interview project for a job you want?

Take-home projects are a great opportunity to show your skills. If possible, I would ask if there's a work trial before inventing your own non-solicited project.

My GWWC donations: Switching from long- to near-termist opportunities?

This makes a lot of sense to me. Personally I'm trying to use my career to work on longtermism, but focusing my donations on global poverty. A few reasons, similar to what you outlined above:

  • I don't want to place all my bets on longtermism. I'm sufficiently skeptical of arguments about AI risk, and sufficiently averse to pinning all my personal impact on a low-probability high-EV cause area, that I'd like to do some neartermist good with my life. Also, this
  • Comparatively speaking, longtermism needs more people and global poverty needs more cash. Give
... (read more)

I came to say the same thing. I was (not that long ago) working on longtermist stuff and donating to neartermist stuff (animal welfare). I think this is not uncommon among people I know.

Free-spending EA might be a big problem for optics and epistemics

There are lots of ways to accurately predict a job applicant’s future success. See the meta-analysis linked below, which finds general mental ability tests, work trials, and structured interviews all to be more predictive of future overall job performance than unstructured interviews, peer ratings, or reference checks.

I’m not a grantmaker and there are certainly benefits to informal networking-based grants, but on the whole I wish EA grantmaking relied less on social connections to grantmakers and more on these kinds of objective evaluations.

Meta-analysis ... (read more)

How much current animal suffering does longtermism let us ignore?

Strongly agreed, and I think it’s one of the most important baseline arguments against AI risk. See Linch’s motivated reasoning critique of effective altruism:

I agree that theorizing is more fun than agonizing (for EA types), but I feel like the counterfactual should be theorizing vs theorizing, or agonizing vs agonizing.

Theorizing: Speaking for myself, I bounced off of both AI safety and animal welfare research, but I didn't find animal welfare research less intellectually engaging, nor less motivating, than AI safety research. If anything the tractability and sense of novel territory makes it more motivating. Though maybe I'd find AI safety research more fun if I'm better at math. (I'm doing my current researc... (read more)

Will FTX Fund publish results from their first round?

Tyler Cowen’s Emergent Ventures fast grants program also releases the funded project with a short summary of their work. Seems like a very good idea, though maybe not the highest priority for the FTX team right now.

Why not offer a multi-million / billion dollar prize for solving the Alignment Problem?

Yeah that's a good point. Another hack would be training a model on text that specifically includes the answers to all of the TruthfulQA questions. 

The real goal is to build new methods and techniques that reliably improve truthfulness over a range of possible measurements. TruthfulQA is only one such measurement, and performing well on it does not guarantee a signficant contribution to alignment capabilities. 

I'm really not sure what the unhackable goal looks like here. 

My colleagues have often been way too nice about reading group papers, rather than the opposite. (I’ll bet this varies a ton lab-to-lab.)
aogara's Shortform

Fun fact: For 20 years at the peak of the Cold War, the US nuclear launch code was “00000000”.

…Are you freaking kidding me??? EAs at the top level of DOE, please!

H/t: Gavin Leech

Why not offer a multi-million / billion dollar prize for solving the Alignment Problem?

For example, TruthfulQA is a quantitative benchmark for measuring the truthfulness of a language model. Achieving strong performance on this benchmark would not alone solve the alignment problem (or anything close to that), but it could potentially offer meaningful progress towards the valuable goal of more truthful AI.

This could be a reasonable benchmark for which to build a small prize, as well as a good example of the kinds of concrete goals that are most easily incentivized.

Here’s the paper:

I like the TruthfulQA idea/paper a lot, but I think incentivizing people to optimize against it probably wouldn't be very robust, and non-alignment-relevant ideas could wind up making a big difference. Just one of several issues: The authors selected questions adversarially against GPT-3—i.e., they oversampled the exact questions GPT-3 got wrong—so, simply replacing GPT-3 with something equally misaligned but different, like Gopher, should yield significantly better performance. That's really not something you want to see in an alignment benchmark.
Why not offer a multi-million / billion dollar prize for solving the Alignment Problem?
Answer by aogaraApr 17, 202218

The main challenge seems to be formulating the goal in a sufficiently specific way. We don’t currently have a benchmark that would serve as a clear indicator of solving the alignment problem. Right now, any proposed solution ends up being debated by many people who often disagree on the solution’s merits.

FTX Future Fund listed AI Alignment Prizes on their ideas page and would be interested in funding them. Given that, it seems like coming up with clear targets for AI safety research would be very impactful.

My solution to this problem (originally posted here) is to run builder/breaker tournaments:

  • People sign up to play the role of "builder", "breaker", and/or "judge".
  • During each round of the tournament, triples of (builder, breaker, judge) are generated. The builder makes a proposal for how to build Friendly AI. The breaker tries to show that the proposal wouldn't work. ("Builder/breaker" terminology from this report.) The judge moderates the discussion.
    • Discussion could happen over video chat, in a Google Doc, in a Slack channel, or whatever. Personall
... (read more)
7Harrison Durland1mo
^ I am not super familiar with the history of “solve X problem and win Y reward,” but my casual familiarity/memory only can think of examples where a solution was testable and relatively easy to objectively specify. With the alignment problem, it seems plausible that some proposals could be found to likely “work” in theory, but getting people to agree on the right metrics seems difficult and if it goes poorly we might all die.
For example, TruthfulQA is a quantitative benchmark for measuring the truthfulness of a language model. Achieving strong performance on this benchmark would not alone solve the alignment problem (or anything close to that), but it could potentially offer meaningful progress towards the valuable goal of more truthful AI. This could be a reasonable benchmark for which to build a small prize, as well as a good example of the kinds of concrete goals that are most easily incentivized. Here’s the paper: []
The Effective Institutions Project is hiring

Fantastic to see such strong progress on the institutional decision making front. Hoping that all goes well, and that EA’s newfound riches might even enable better funding for your hiring plans.

aogara's Shortform

Collected Thoughts on AI Safety

Here are of some of my thoughts on AI timelines:

And here are some thoughts on other AI Safety topics:

Generally speaking, I believe in longer timelines ... (read more)

How effective is sending a pre-interview project for a job you want?

Of course Warren, hope it’s helpful! I had a strong sense of what each company was looking for before investing time in a project. Usually this was from talking with them first, though in the case of AI Impacts it came from a public call for collaborators on the 80K podcast. I also always submit a normal job application, and usually I would only do a work project after speaking with someone and learning what they’re looking for, which usually comes from the application. (When I have a dream job that I know a ton about, then I’m more inclined to take the ti... (read more)

Thanks for expanding! I know that some hiring processes in tech involve take home projects so I’m wondering how that played out if you had any of those despite doing a non-solicited project for them already?
aogara's Shortform

Update on Atlas Fellowship: They've extended their application period by one week! Good decision for getting more qualified applications into the pipeline. I wonder how many applications they've received overall. 

How effective is sending a pre-interview project for a job you want?

I got a four month work trial at AI Impacts after spending ~20 hours on an unsolicited pre-interview project, parts of which were later published on their website. I’m not sure if I would’ve gotten the interview otherwise; I was an undergraduate with no experience in AI at the time.

20 hours is definitely overkill, but in general, my goal in interviews and work trials is to ask lots of specific questions about what the employer needs and figure out how I can provide it. You can describe their problem and your specific skills in a PowerPoint or simply in you... (read more)

Thanks for sharing your experience! A couple of follow-up questions: * Did you send this unsolicited pre-interview project without talking to anyone at these companies? What were the responses like? * How did the results change between targeting small companies vs larger companies? * To clarify, you only sent an email with your pre-interview project without submitting an application?
Effective data science projects

Hey, I think this is a great idea. Credo AI is an organization working on data science-type projects for AI safety, maybe one of their projects could give you inspiration?

aogara's Shortform

Hey Evan, these are definitely stronger points against short timelines if you believe in slow takeoff, rather than points against short-timelines in a hard takeoff world. It might come as no surprise that I think slow takeoff is much more likely than hard takeoff, with the Comprehensive AI Systems model best representing what I would expect. A short list of the key arguments there:

... (read more)
aogara's Shortform

Concerns with BioAnchors Timelines

A few points on the Bio Anchors framework, and why I expect TAI to require much more compute than used by the human brain:

1. Today we routinely use computers with as much compute as the human brain. Joe Carlsmith’s OpenPhil report finds the brain uses between 10^13 and 10^17 FLOP/s. He points out that Nvidia's V100 GPU retailing for $10,000 currently performs 10^14 FLOP/s. 

2. Ajeya Cotra’s Bio Anchors report shows that AlphaStar's training run used 10^23 FLOP, the equivalent of running a human brain-sized computer wit... (read more)

aogara's Shortform

I strongly disagree with the claim that there is a >10% chance of TAI in the next 10 years. Here are two small but meaningful pieces of why I have much longer AI timelines. 

Note that TAI is here defined as one or both of: (a) any 5 year doubling of real global GDP, or (b) any catastrophic or existential AI failures.  

Market Activity

Top tech companies do not believe that AI takeoff is around the corner. Mark Zuckerberg recently saw many of his top AI research scientists leave the company, as Facebook has chosen to acquire Oculus and bet on the ... (read more)

These are thoughtful data points, but consider that they may just be good evidence for hard takeoff rather than soft takeoff.

What I mean is that most of these examples show a failure of narrow AIs to deliver on some economic goals. In soft takeoff, we expect to see things like broad deployment of AIs contributing to massive economic gains and GDP doublings in short periods of time well before we get to anything like AGI.

But in hard takeoff, failure to see massive success from narrow AIs could happen due to regulations and other barriers (or it could just b... (read more)

The Vultures Are Circling

Would really appreciate links to Twitter threads or any other publicly available versions of these conversations. Appreciate you reporting what you’ve seen but I haven’t heard any of these conversations myself.

I sent a DM to the author asking if they could share examples. If you know of any, please DM me!

Yes to links of what conversations on gaming the system are happening where! 

Surely this is something that should be shared directly with all funders as well? Are there any (in)formal systems in place for this?

aogara's Shortform

Appreciate and agree with both of these comments. I’ve made a brief update to the original post to reflect it, and hope to respond in more detail soon.

aogara's Shortform

The investment in advertising, versus the consumption-style spending on GiveDirectly? Just meant to compare the impact of the two. The first’s impact would come by raising more money to eventually be donated, the second is directly impactful, so I’d like to think about which is a better use of the funds.

I feel anxious that there is all this money around. Let's talk about it

Thanks for the corrections, fixed. I agree that the hits-based justification could work out, just would like to see more public analysis of this and other FTX initiatives.

aogara's Shortform

Some thoughts on FTX copied from this thread:

One way to approach this would simply be to make a hypothesis (i.e. the bar for grants is being lowered, we're throwing money at nonsense grants), and then see what evidence you can gather for and against it.

Thinking about FTX and their bar for funding seems very important. I'm thrilled that so much money is being put towards EA causes, but a few early signs have been concerning. Here’s two considerations on the hypothesis that FTX has a lower funding bar than previous EA funding.

First, it seems that FTX would l... (read more)

Update on Atlas Fellowship: They've extended their application period by one week []! Good decision for getting more qualified applications into the pipeline. I wonder how many applications they've received overall.

First, it seems that FTX would like to spend a lot more a lot faster than has been the EA consensus for a long time. ... It also strikes against recent work on patient philanthropy, which is supported by Will MacAskill's argument that we are not living in the most influential time in human history. 

I don't think fast spending in and of itself strikes against patient longtermism: see Owen-Cotton-Barratt's post "Patient vs urgent longtermism" has little direct bearing on giving now vs later.

5Chris Leong2mo
I suggest caution with trying to compare the future fund's investments against the donating to global poverty without engaging with the long-termist worldview. This worldview you could be right or wrong but it is important to engage with it to understand why FTX might consider these investments worthwhile. Another part of the argument is that there is currently an absurd amount of money per effective altruist. This might not matter for global poverty where much of the work can be outsourced, but it is a much bigger problem for many projects in other areas. In this scenario, it might make sense to exchange apps to seeming amounts of money to grow the pool of committed members, at least if this really is the bottleneck, particularly if you believe that certain projects need to be completed on short timelines. I agree being situated in the Bahamas is less than deontologically spotless but I don't believe that avoiding the negative PR is worth billions of dollars and I don't see it as a particularly egregious moral violation nor do I see this as significantly reducing trust in EA or FTX.
The second sentence seems to be confusing investment with consumption.

It also strikes against recent work on patient philanthropy, which is supported by Will MacAskill's argument that we are not living in the most influential time in human history.


Note that patient philanthropy includes investing in resources besides money that will allow us to do more good later; e.g. the linked article lists "global priorities research" and "Building a long-lasting and steadily growing movement" as promising opportunities from a patient longtermist view.

Looking at the Future Fund's Areas of Interest, at least 5 of the 10 strike me as... (read more)

Two factual nitpicks:

  1. The fellowship's $50k to 100 fellows, a total of $5.5mil.

2. The money's not described by AF as "no strings attached." From their FAQ:

Scholarship money should be treated as “professional development funding” for award winners. This means the funds could be spent on things like professional travel, textbooks, technology, college tuition, supplementing unpaid internships, and more.

Students will receive ongoing guidance to manage and effectively spend their scholarship funds.

For Fellows ($50,000), a (taxed) amount is placed in a trust fund

... (read more)
5Rockwell Schwartz2mo
FYI, this just links to this same Forum post for me.
Critique of OpenPhil's macroeconomic policy advocacy

Would you say that OpenPhil's grants in 2021 were negative impact, but that many of their previous grants were positive impact? You demonstrate quite convincingly that the 2021 grants were negative impact (if they had impact at all), pushing us from supporting employment and consumption post-Covid to triggering inflation. But OpenPhil's macroeconomic policy grants date back to 2014, when the case for more expansionary monetary policy was much stronger. 

The monetary consensus was significantly more hawkish during the recovery from the 2008 recession. T... (read more)

My vague sense is that you're right and that until 2021, the program was very good and beneficial and helped with a faster recovery. I commend OpenPhil for engaging with this and I agree that we should evaluate the impact of the whole program, I just don't have the capacity and means to do that. My vague sense is that the program would come out net positive on the whole, however, perhaps if there are large negative effects of current high inflation - like lots of populists getting elected and we can causally attribute this to high inflation-  then it ... (read more)

$1 to extend an infant's life by one day?
Answer by aogaraMar 28, 202216

Hey, glad you're interested in donating! 

The Against Malaria Foundation distributes bednets to protect people from being bitten by mosquitos that carry malaria, a deadly disease that kills hundreds of thousands of people each year. GiveWell provides a cost-effectiveness analysis based on their extensive research on the charity and, while they strongly caution against taking these estimates as fact, their estimate is that $1 will extend an infant's life by at least two days. (Calculation: 60 years of additional life * 365 days / $9,064 maximum cost per... (read more)

Aogara, What a perfect reply! This thoughtful estimate of Against Malaria Foundation expected benefit thoroughly informed me. The similar expectations from Malaria Consortium, New Incentives, Helen Keller International, and GiveWell's Maximum Impact Fund have sound evidence to extend the healthy life of an infant, on average, by about one day for one dollar.
Meditations on careers in AI Safety

+1 to all of this. Sounds like a very tough decision. If it were me, I would probably choose quality of life and stick with the startup. (Might also donate to areas that are more funding constrained like global development and animal welfare.)

Thanks for making concrete bets @aogara :)
How To Become A Professional Software Developer

Nice post. One suggestion for an area of specialization: web development. Building front-ends and back-ends for websites seems like one of the areas of software engineering with the hottest hiring market and easiest bar for entry. Many coding bootcamps focus on teaching web development skills during a 12 week course, and then immediately recommend applicants apply for a job. If you can build a nice looking full-stack webapp with an API connection to a backend database that's hosted on e.g. Heroku and has the full code visible on GitHub, you have a good chance of being hired as a web developer. From there, you can branch into many other fields of software engineering. 

7Yonatan Cale2mo
I agree with everything you said about web/fullstack development (upvoted!), I'd just like to push back on "hottest hiring market" as an important consideration. I know this may be controversial, hear me out: It is pretty hard to pick an area of software engineering today where it will be hard to find a job. Picking a slightly hotter area won't make much difference. (And worse: It's a question of supply and demand. Fullstack devs are also relatively common). Anyway, my point is that I think this consideration is over rated, and more importantly, distracts some people from something else: What's underrated in my opinion? (Within EA) Personal fit within software development. This is worthy of an entire post, I think. TL;DR: 1. My usefulness as a developer is very much effected by my skill. 2. The speed I build skill is very much effected by how much my job interests me. (Please remember this is only a TL;DR) My disclaimer would be "if you think you're going to chose a subdomain of software where there is too little demand, feel free to ask about it". For example, I wouldn't recommend learning Pascal. But in practice from the actual conversations I had with EA devs, none of them aimed in a direction I thought was bad. Still, fullstack is great, has market demand, has EA demand, and in my personal opinion is very fun as well
Milan Griffes on EA blindspots

Hey Peter, on your last point, I believe the clearest paths from AI to x-risk run directly through either nuclear weapons or bioweapons. Not sure if the author believes the same, but here’s some thoughts I wrote up on the topic:

Yes, I have a similar position that early-AGI risk runs through nuclear mostly.  I wrote my thoughts on this here: When Bits Split Atoms

2Peter Wildeford2mo
Thanks I'll take a look!
Milan Griffes on EA blindspots

Yeah, understandable but I would also push back. Mining / buying your own uranium and building a centrifuge to enrich it and putting it into a missile is difficult for even rogue nations like Iran. An advanced AI system might just be lines of code in a computer that can use the internet and output text or speech, but with no robotics system to give it physical capacity. From that point of view, building your own nukes seems much more difficult than hacking into an existing ICBM system.

The Future Fund’s Project Ideas Competition

Strongly agree with this. There are only a handful of weapons that threaten catastrophe to Earth’s population of 8 billion. When we think about how AI could cause an existential catastrophe, our first impulse shouldn’t be to think of “new weapons we can’t even imagine yet”. We should secure ourselves against the known credible existential threats first.

Wrote up some thoughts about doing this as a career path here:

Load More