All of aogara's Comments + Replies

“ People are often grateful to you for granting them money. This is a mistake.”

How would you recommend people react when they receive a grant? Saying thank you simply seems polite and standard etiquette, but I agree that it misportrays the motives of the grantmaker and invites concerns of patronage and favoritism.

Really cool! I was hoping to attend but had to be home for a family event. Would be super interested to see any participants summarize their thoughts on AI timelines, or a poll of the group's opinions. 

Sounds like Decision Transformers (DTs) could quickly become powerful decision-making agents. Some questions about them for anybody who's interested: 

DT Progress and Predictions

Outside Gato, where have decision transformers been deployed? Gwern shows several good reasons to expect that performance could quickly scale up (self-training, meta-learning, mixture of experts, etc.). Do you expect the advantages of DTs to improve state of the art performance on key RL benchmark tasks, or are the long-term implications of DTs more difficult to measure? Focusi... (read more)

Cool arguments on the impact of policy work for AI safety. I find myself agreeing with Richard Ngo’s support of AI policy given the scale of government influence and the uncertain nature of AI risk. Here’s a few quotes from the piece.

How AI could be influenced by policy experts:

in a few decades (assuming long timelines and slow takeoff) AIs that are less generally intelligent that humans will be causing political and economic shockwaves, whether that's via mass unemployment, enabling large-scale security breaches, designing more destructive weapons, psycho

This is a terrific distillation, thanks for sharing! I really like the final three sections with implications for short-term, long-term, and policy risks. 

For example, in 2019 the U.S. Food and Drug Administration issued a proposed regulatory framework for AI/ML-based software used in health care settings. Less than a week ago, the U.S. Justice Department and the Equal Employment Opportunity Commission released guidance and technical assistance documents around avoiding disability discrimination when using AI for hiring decisions.

These are some great ... (read more)

I agree, upvotes do seem a bit inflated. It creates an imbalance between new and old users that continually grows as existing users rack up more upvotes over time. This can be good for preserving culture and norms, but as time goes on, the difference between new and old users only grows. Some recalibration could help make the site more welcoming to new users.

In general, I think it would be nice if each upvote counted for roughly 1 karma. Will MacAskill’s most recent post received over 500 karma from only 250 voters, which might exaggerate the reach of the... (read more)

Hey Aman, thanks for the post. It does seem a bit outdated that the top picture for altruism  is a French painting from hundreds of years ago. EA should hope to change the cultural understanding of doing good from something that's primarily religious or spiritual, to something that can be much more scientific and well-informed. 

I do think some of the accusations of EA being a cult might go a bit deeper. There aren't many other college clubs that would ask you to donate 10% of your income or determine your career plans based on their principles. O... (read more)

This is a great point. As one example of growing mainstream coverage, here’s a POLITICO Magazine piece on Carrick Flynn’s Congressional campaign. It gives a detailed explanation of effective altruism and longtermism, and seems like a great introduction to the movement for somebody new. The author sounds like he might have collaborated with CEA, but if not, maybe someone should reach out?

https://www.politico.com/news/magazine/2022/05/12/carrick-flynn-save-world-congress-00031959

Coverage of this post from The Hill on April 24th:  

Many of Flynn’s donors are involved in an online forum called Effective Altruism, a group that analyzes how best to spend money on philanthropic efforts. Their conclusion, according to some of the posts backing Flynn, has been that spending a few million on a congressional race could result in billions in spending on pandemic preparedness by the federal government.

Flynn is “the first person to ever run for US congress on a platform of preventing future pandemics,” wrote one user, Andrew Snyder-Beatti

Donated because of this post. Thanks for sharing and good luck to Carrick.

Hey, this is a great question with good context for potential answers too. If you don’t get any substantive responses here, I’d suggest posting as a question on the frontpage — the shortforms really don’t get much visibility.

Great overview of an important field for AI safety, thanks for sharing. A few questions if you have the time:

First, what secrets would be worth keeping? Most AI research today is open source, with methods described in detailed papers and code released on GitHub. That which is not released is often quickly reverse-engineered: OpenAI’s GPT-3 and DALLE-2 systems, for example, both have performant open-source implementations. On the other hand, many government and military applications seemingly must be confidential. 

What kinds of AI research is kept secr... (read more)

Thanks for sharing. It seems like the most informed people in AI Safety have strongly changed their views on the impact of OpenAI and Deepmind compared to only a few years ago. Most notably, I was surprised to see ~all of the OpenAI safety team leave for Anthropic . This shift and the reasoning behind it have been fairly opaque to me, although I try to keep up to date. Clearly there are risks with publicly criticizing these important organizations, but I'd be really interested to hear more about this update from anybody who understands it.

Very important perspective from someone on the front lines of recruiting new EAs. Thanks for sharing!

Take-home projects are a great opportunity to show your skills. If possible, I would ask if there's a work trial before inventing your own non-solicited project.

Hi, would Anthropic's research agenda be a good candidate for distilling?

This makes a lot of sense to me. Personally I'm trying to use my career to work on longtermism, but focusing my donations on global poverty. A few reasons, similar to what you outlined above:

• I don't want to place all my bets on longtermism. I'm sufficiently skeptical of arguments about AI risk, and sufficiently averse to pinning all my personal impact on a low-probability high-EV cause area, that I'd like to do some neartermist good with my life. Also, this. 
• Comparatively speaking, longtermism needs more people and global poverty needs more cash. Give

There are lots of ways to accurately predict a job applicant’s future success. See the meta-analysis linked below, which finds general mental ability tests, work trials, and structured interviews all to be more predictive of future overall job performance than unstructured interviews, peer ratings, or reference checks.

I’m not a grantmaker and there are certainly benefits to informal networking-based grants, but on the whole I wish EA grantmaking relied less on social connections to grantmakers and more on these kinds of objective evaluations.

Meta-analysis ... (read more)

Strongly agreed, and I think it’s one of the most important baseline arguments against AI risk. See Linch’s motivated reasoning critique of effective altruism:

https://forum.effectivealtruism.org/posts/pxALB46SEkwNbfiNS/the-motivated-reasoning-critique-of-effective-altruism

Tyler Cowen’s Emergent Ventures fast grants program also releases the funded project with a short summary of their work. Seems like a very good idea, though maybe not the highest priority for the FTX team right now.

Yeah that's a good point. Another hack would be training a model on text that specifically includes the answers to all of the TruthfulQA questions. 

The real goal is to build new methods and techniques that reliably improve truthfulness over a range of possible measurements. TruthfulQA is only one such measurement, and performing well on it does not guarantee a signficant contribution to alignment capabilities. 

I'm really not sure what the unhackable goal looks like here. 

Fun fact: For 20 years at the peak of the Cold War, the US nuclear launch code was “00000000”.

…Are you freaking kidding me??? EAs at the top level of DOE, please!

https://gizmodo.com/for-20-years-the-nuclear-launch-code-at-us-minuteman-si-1473483587

H/t: Gavin Leech

For example, TruthfulQA is a quantitative benchmark for measuring the truthfulness of a language model. Achieving strong performance on this benchmark would not alone solve the alignment problem (or anything close to that), but it could potentially offer meaningful progress towards the valuable goal of more truthful AI.

This could be a reasonable benchmark for which to build a small prize, as well as a good example of the kinds of concrete goals that are most easily incentivized.

Here’s the paper: https://arxiv.org/pdf/2109.07958.pdf

Fantastic to see such strong progress on the institutional decision making front. Hoping that all goes well, and that EA’s newfound riches might even enable better funding for your hiring plans.

Collected Thoughts on AI Safety

Here are of some of my thoughts on AI timelines:

• Who believes what about AI timelines
• Why I have longer timelines than the BioAnchors report
• Reasons to prefer slow takeoff speeds
• Why market activity and data constraints lengthen my timelines
• Three scenarios for AI progress

And here are some thoughts on other AI Safety topics:

• Questions about Decision Transformers and Deepmind's Gato
• Why AI policy seems valuable (selected quotes from Richard Ngo)
• Why AI alignment prizes are so difficult

Generally speaking, I believe in longer timelines ... (read more)

Of course Warren, hope it’s helpful! I had a strong sense of what each company was looking for before investing time in a project. Usually this was from talking with them first, though in the case of AI Impacts it came from a public call for collaborators on the 80K podcast. I also always submit a normal job application, and usually I would only do a work project after speaking with someone and learning what they’re looking for, which usually comes from the application. (When I have a dream job that I know a ton about, then I’m more inclined to take the ti... (read more)

Update on Atlas Fellowship: They've extended their application period by one week! Good decision for getting more qualified applications into the pipeline. I wonder how many applications they've received overall. 

Hey Evan, these are definitely stronger points against short timelines if you believe in slow takeoff, rather than points against short-timelines in a hard takeoff world. It might come as no surprise that I think slow takeoff is much more likely than hard takeoff, with the Comprehensive AI Systems model best representing what I would expect. A short list of the key arguments there:

• Discontinuities on important metrics are rare, see the AI Impacts writeup. EDIT: Dan Hendrycks and Thomas Woodside provide a great empirical survey of AI progress across several

Concerns with BioAnchors Timelines

A few points on the Bio Anchors framework, and why I expect TAI to require much more compute than used by the human brain:

1. Today we routinely use computers with as much compute as the human brain. Joe Carlsmith’s OpenPhil report finds the brain uses between 10^13 and 10^17 FLOP/s. He points out that Nvidia's V100 GPU retailing for $10,000 currently performs 10^14 FLOP/s. 

2. Ajeya Cotra’s Bio Anchors report shows that AlphaStar's training run used 10^23 FLOP, the equivalent of running a human brain-sized computer wit... (read more)

I strongly disagree with the claim that there is a >10% chance of TAI in the next 10 years. Here are two small but meaningful pieces of why I have much longer AI timelines. 

Note that TAI is here defined as one or both of: (a) any 5 year doubling of real global GDP, or (b) any catastrophic or existential AI failures.  

Market Activity

Top tech companies do not believe that AI takeoff is around the corner. Mark Zuckerberg recently saw many of his top AI research scientists leave the company, as Facebook has chosen to acquire Oculus and bet on the ... (read more)

Would really appreciate links to Twitter threads or any other publicly available versions of these conversations. Appreciate you reporting what you’ve seen but I haven’t heard any of these conversations myself.

Appreciate and agree with both of these comments. I’ve made a brief update to the original post to reflect it, and hope to respond in more detail soon.

The investment in advertising, versus the consumption-style spending on GiveDirectly? Just meant to compare the impact of the two. The first’s impact would come by raising more money to eventually be donated, the second is directly impactful, so I’d like to think about which is a better use of the funds.

Thanks for the corrections, fixed. I agree that the hits-based justification could work out, just would like to see more public analysis of this and other FTX initiatives.

Thanks, fixed.

Some thoughts on FTX copied from this thread:

One way to approach this would simply be to make a hypothesis (i.e. the bar for grants is being lowered, we're throwing money at nonsense grants), and then see what evidence you can gather for and against it.

Thinking about FTX and their bar for funding seems very important. I'm thrilled that so much money is being put towards EA causes, but a few early signs have been concerning. Here’s two considerations on the hypothesis that FTX has a lower funding bar than previous EA funding.

First, it seems that FTX would l... (read more)

Moved this comment to a shortform post here.

Would you say that OpenPhil's grants in 2021 were negative impact, but that many of their previous grants were positive impact? You demonstrate quite convincingly that the 2021 grants were negative impact (if they had impact at all), pushing us from supporting employment and consumption post-Covid to triggering inflation. But OpenPhil's macroeconomic policy grants date back to 2014, when the case for more expansionary monetary policy was much stronger. 

The monetary consensus was significantly more hawkish during the recovery from the 2008 recession. T... (read more)

+1 to all of this. Sounds like a very tough decision. If it were me, I would probably choose quality of life and stick with the startup. (Might also donate to areas that are more funding constrained like global development and animal welfare.)

Nice post. One suggestion for an area of specialization: web development. Building front-ends and back-ends for websites seems like one of the areas of software engineering with the hottest hiring market and easiest bar for entry. Many coding bootcamps focus on teaching web development skills during a 12 week course, and then immediately recommend applicants apply for a job. If you can build a nice looking full-stack webapp with an API connection to a backend database that's hosted on e.g. Heroku and has the full code visible on GitHub, you have a good chance of being hired as a web developer. From there, you can branch into many other fields of software engineering. 

Check it out! Critique of OpenPhil's macroeconomic policy advocacy

Hey Peter, on your last point, I believe the clearest paths from AI to x-risk run directly through either nuclear weapons or bioweapons. Not sure if the author believes the same, but here’s some thoughts I wrote up on the topic:

https://forum.effectivealtruism.org/posts/7ZZpWPq5iqkLMmt25/aidan-o-gara-s-shortform?commentId=rnM3FAHtBpymBsdT7

Yeah, understandable but I would also push back. Mining / buying your own uranium and building a centrifuge to enrich it and putting it into a missile is difficult for even rogue nations like Iran. An advanced AI system might just be lines of code in a computer that can use the internet and output text or speech, but with no robotics system to give it physical capacity. From that point of view, building your own nukes seems much more difficult than hacking into an existing ICBM system.

Strongly agree with this. There are only a handful of weapons that threaten catastrophe to Earth’s population of 8 billion. When we think about how AI could cause an existential catastrophe, our first impulse shouldn’t be to think of “new weapons we can’t even imagine yet”. We should secure ourselves against the known credible existential threats first.

Wrote up some thoughts about doing this as a career path here: https://forum.effectivealtruism.org/posts/7ZZpWPq5iqkLMmt25/aidan-o-gara-s-shortform?commentId=rnM3FAHtBpymBsdT7