All of eca's Comments + Replies

Try working on something random with someone cool

One impression I could imagine having after reading this post for the first time is something like: "eca would prefer fewer connections to people and doesn't value that output of community building work" or even more scandalously, "eca thinks community builders are wasting their time".

I don't believe that, and would have edited the draft to make that more clear if I had taken a different approach to writing it.

A quick amendment to that vibe.

1. Community building is mission critical. It's also complicated, and not something I expect to have good opinions abo
Try working on something random with someone cool

Meta note: this was an experiment in jotting something down. I've had a lot of writers block on forum posts before and thought it would be good to try erring on the side of not worrying about the details.

As I'm rereading what I wrote late last night I'm seeing things I wish I could change. If I have time, I'll try writing these changes as comments rather than editing the post (except for minor errors).

(Curious for ideas/approaches/ recommendations for handling this!)

Try working on something random with someone cool

This seems like a great idea- I actually woke up this morning realizing I forgot it from my list!

One part of my perspective which is possibly worth reemphasizing: IMO, what you choose to work together does not need to be highly optimized or particularly EA. At least to make initial progress in this direction, it seems plausible that you should be happy with anything challenging/ without an existing playbook, collaborative, and “real” in the sense of requiring you to act like you would if you were solving a real problem instead of playing a toy game.

Concrete Biosecurity Projects (some of which could be big)

thanks for the kind words! I agree that we didn't have much good stuff for ppl to do 4 yrs ago when i started in bio but don't feel like my model matches yours regarding why.

But I'm also wanting to confirm I've understood what you are looking for before I ramble.

How much would you agree with this description of what I could imagine filling in from what you said re 'why it took so long':

"well I looked at this list of projects, and it didn't seem all that non-obvious to me, and so the default explanation of 'it just took a long time to work out these project... (read more)

8Ozzie Gooen7mo
That sounds like much of it. To be clear, it's not that the list is obvious, but more that it seems fairly obvious that a similar list was possible. It seemed pretty clear to me a few years ago that there must be some reasonable lists of non-info-hazard countermeasures that we could work on, for general-purpose bio safety. I didn't have these particular measures in mind, but figured that roughly similar ones would be viable. Another part of my view is, "Could we have hired a few people to work full-time coming up with a list about this good, a few years earlier?" I know a few people who were discouraged from working in the field earlier on because their was neither the list, nor the go-ahead to try to make a list.
Concrete Biosecurity Projects (some of which could be big)

meta note- its super cool to see all this activity! but the volume is makin me a bit stressed and i probably won't be trying to respond to lots even if i do one sporadically. does not mean i am ignoring you!

Well I hope it works out for ya! Thanks haha

In case you are looking for content and have interests similar to me I like the following for audio:

• Institute for Advanced Study lectures (random fun science)
• Yannic Kilcher (ML paper summaries)
• Wendover Productions/ Kurzgesagt (random probably not as useful but interesting science and econ funfacts)
• LiveOverflow (Security)

And i find that searching for random academics names is more likely to turn up lectures/ convos than podcasts

4BrianTan9mo
Thanks for the recommendations! I don't think I'll listen to these, though Wendover Productions looks cool. I might try listening to some EA Global videos or GPI lectures via audio on YouTube premium.
I’ll pay you a $1,000 bounty for coming up with a good bounty (x-risk related) Are you looking for shovel ready bounties (eg write them up and you are good to go) or things which might need development time (eg figuring out exactly what to reward, working out the strategy of why the bounty might be good etc)? Shovel ready bounties are preferred but to avoid premature exploitation I'd just like to hear as many ideas as possible at this point. Some ideas might require back and forth, but that's ok! Seeing the ideas coming in is already giving me lots of ideas for ways to potentially scale this. Can EA leverage an Elon-vs-world-hunger news cycle? FWIW this seems like a reasonable idea to me and I would be pretty sad if no one at e.g. Givewell had even considered it. What are your favourite ways to buy time? Answer by ecaNov 02, 202170 • Order groceries online! Maybe this is obvious but I have the impression not as many ppl do this as they should. Saves me at least 1 hr (usually closer to 2) for <$20
• Pay for a bunch of disk space. I find it generates a lot of overhead to have files in different places. For me, the solution has been a high performance workstation plus remote desktop forwarding to my laptop when I travel so I can always have the same disk and workspace
• Buy more paid apps/ premium upgrades/ digital subscriptions. I haven’t done the math on this so might not be as good as
3Quinn McHugh (he/him)9mo
+1 for ordering groceries online - Tried this for the first time last week. For me, not having to expend the time & mental energy searching for the things I need was well worth the small financial cost. For those in the US and Canada who still like to get out of the house, Instacart offers grocery pickup [https://www.instacart.com/help/section/360007902831/360039162212].
5BrianTan9mo
eca's Shortform

Quest: see the inside of an active bunker

1N N3mo
Why, if you don't mind me asking?
Listen to more EA content with The Nonlinear Library

Seems like a good idea if it were easy

Listen to more EA content with The Nonlinear Library

(Sorry, when I said your story for impact was "plausible", in my head I was comparing it to my own idea for why this would be good, and I meant that it was plausibly better than my story. I actually buy your pitch as written, seems like a solidly good thing; apologies)

Listen to more EA content with The Nonlinear Library

What a cool project! I listen to the vast majority of my reading these days and am perpetually out of good things to read.

The linked audio is reasonably high quality, and more importantly, it doesn't have some of the formatting artifacts that other TTS programs have. Well done.

Your story for why this is a potentially high impact project is plausible to me, especially given how much you've automated. I have independently been thinking about building something similar, but with a very different story for why it could be worth my time to do it. That means th... (read more)

2eca10mo
(Sorry, when I said your story for impact was "plausible", in my head I was comparing it to my own idea for why this would be good, and I meant that it was plausibly better than my story. I actually buy your pitch as written, seems like a solidly good thing; apologies)
Listen to more EA content with The Nonlinear Library

And there are various things one could probably do to make it not illegal but still messed up and the wrong thing to do! Like make it mandatory to check a box saying you waive your copyright for audio on a thing before you post on the forum. I think if, like some of the tech companies, you made this box really little and hard to find, most people would not change their posting behavior very much, and would now be totally legal (by assumption).

but it would still be a bad thing to do.

Listen to more EA content with The Nonlinear Library

This is a reason to fix the system! My point is that it reduces to "make all the authors happy with how you are doing things", there is not some spooky extra thing having to do with illegality

TBC I do not endorse using people's content in a way they aren't happy with, but I would still have that same belief if it wasn't illegal at all to do so.

Listen to more EA content with The Nonlinear Library

I use speechify, its voices are quite good but has the same formatting issues as all the rest (reading junk text) which I think is the real bottleneck here

Listen to more EA content with The Nonlinear Library

FWIW I think I endorse Kat's reasoning here. I don't think it matters if it is illegal if I'm correct in suspecting that the only people who could bring a copyright claim are the authors, and assuming the authors are happy with the system being used. This is analogous to the way it is illegal, by violating minimum wage laws, to do work for your own company without paying yourself, but the only person who has standing to sue you is AFAIK yourself.

Not a lawyer, not claiming to know the legal details of these cases, but I think this standing thing is real and an appropriate way to handle

I've seen this reasoning a lot, where EA organisations assume they won't get sued because the only people they're illegally using the data of are other EAs, and and as someone whose data has been misused with this reasoning, I don't love it!

eca's Shortform

Empirical differential tech development?

Many longtermist questions related to dangers from emerging tech can be reduced to “what interventions would cause technology X to be deployed before/ N years earlier than/ instead of technology Y”.

In, biosecurity, my focus area, an example of this would be something like "how can we cause DNA synthesis screening to be deployed before desktop synthesizers are widespread?"

It seems a bit cheap to say that AI safety boils down to causing an aligned AGI before an unaligned, but it kind of basically does, and I suspect ... (read more)

My current best guess on how to aggregate forecasts

I wonder how these compare with fitting a Beta distribution and using one of its statistics? I’m imagining treating each forecast (assuming they are probabilities) as an observation, and maximizing the Beta likelihood. The resulting Beta is your best guess distribution over the forecasted variable.

It would be nice to have an aggregation method which gave you info about the spread of the aggregated forecast, which would be straightforward here.

It's not clear to me that "fitting a Beta distribution and using one of it's statistics" is different from just taking the mean of the probabilities.

I fitting a beta distribution to Metaculus forecasts and looked at:

• Median forecast
• Mean forecast
• Mean log-odds / Geometric mean of odds
• Fitted beta median
• Fitted beta mean

Scattering these 5 values against each other I get:

We can see fitted values are closely aligned with the mean and mean-log-odds, but not with the median. (Unsurprising when you consider the ~parametric formula for the mean / median).

7Jaime Sevilla10mo
Hmm good question. For a quick foray into this we can see what would happen if we use our estimate the mean of the max likelihood beta distribution implied by the sample of forecastsp1,...,pN. The log-likelihood to maximize is then logL(α,β)=(α−1)∑ilogpi+(β−1)∑ilog(1−pi)−NlogB(α,β) The wikipedia article on the Beta distribution [https://en.wikipedia.org/wiki/Beta_distribution#Maximum_likelihood] discusses this maximization problem in depth, pointing out that albeit no closed form exists ifαandβcan be assumed to be not too small the max likelihood estimate can be approximated as^α≈12+^GX2(1−^GX−^G1−X)and^β≈12+^G1−X2(1−^GX−^G1−X), whereGX=∏ ip1/NiandG1−X=∏i(1−pi)1/N. The mean of a beta with these max likelihood parameters is^α^α+^β=(1−G1−X)(1−GX) +(1−G1−X). By comparison, the geometric mean of odds estimate is: p=∏Ni=1p1/Ni∏Ni=1p1/Ni+∏Ni=1(1−pi)1/N=GXGX+G1−X Here are two examples of how the two methods compare aggregating five forecasts I originally did this to convince myself that the two aggregates were different. And they seem to be! The method seems to be close to the arithmetic mean in this example. Let's see what happens when we extremize one of the predictions: We have made p3 one hundred times smaller. The geometric mean is suitable affected. The maximum likelihood beta mean stays close to the arithmetic mean, unperturbed. This makes me a bit less excited about this method, but I would be excited about people poking around with this method and related ones!
eca's Shortform

I’m vulnerable to occasionally losing hours of my most productive time “spinning my wheels”: working on sub-projects I later realize don’t need to exist.

Elon Musk gives the most lucid naming of this problem in the below clip. He has a 5 step process which nails a lot of best practices I’ve heard from others and more. It sounds kind of dull and obvious to write down, but somehow I think staring at the steps will actually help. Its also phrased somewhat specifically to building physical stuff, but I think there is a generic version of each. I’m going to try ... (read more)

Open Philanthropy is seeking proposals for outreach projects

One more unsolicited outreach idea while I’m at it: high school career / guidance counselors in the US.

I’m not sure how idiosyncratic this was of my school, but we had this person whose job it was to give advice to older highschool kids about what to do for college and career. Mine’s advice was really bad and I think a number of my friends would have glommed onto 80k type stuff if it was handed to them at this time (when people are telling you to figure out your life all of a sudden). This probably hits the 16yo demographic pretty well.

This could lead to quite a bit of cost-effective positive impact on students, especially those who already have an interest in choosing a career that has positive social consequences. Many students, in my experience, would be very happy to consider higher-impact careers if they had a little wisely-presented encouragement at the right juncture. Such materials would not have to be extensive, and they could be tied to online content that goes deeper into the topic or even provides some interaction. That said, the above OP call for proposals seems highly oriented towards students at elite schools, or elite students at other schools, and specifically is aimed at students heading for a university education. I might suggest that we should be considering how young people likely to enter other professions, be they white- or blue-collar, might benefit from an understanding of these topics (e.g., those listed above: EA, rationality, longtermism, and global catastrophic risk reduction). I will start a discussion on this in the proper forum...but this much larger group of future consumers/workers/influencers/voters should not be ignored. Charities need staff at many levels, and people in many vocations can incorporate these ideas into their work, giving, volunteering, and political activities. Is it too soon for EA to open up to a broader audience?
Open Philanthropy is seeking proposals for outreach projects

Exciting!

This is probably not be the best place to post this but I’ve been learning recently about the success of hacking games in finding and training computer security people (https://youtu.be/6vj96QetfTg for a discussion, also this game I got excited about in high school: https://en.m.wikipedia.org/wiki/Cicada_3301).

I think there might be something to an EA/ rationality game. Like something with a save-the-world but realistically plot and game mechanics built around useful skills like Fermi estimation. This is a random gut feeling I’ve had for a while ... (read more)

4Linch1y
I thought Decision Problem: Paperclips [https://www.decisionproblem.com/paperclips/] introduced a subset of AI risk arguments fairly well in gamified form, but I'm not aware of anybody where the game made them become interested enough in AGI alignment/risk/safety enough to work on it. Does anybody else on this forum have data/anecdata?
How many times would nuclear weapons have been used if every state had them since 1950?

I appreciate the answers so far!

One thing I realized I'm curious about in asking this is something about how many groups of people/ governing bodies are actually crazy enough to use nuclear weapons even if self-annihilation is assured. This seems like an interesting last check against horrible mutual destruction stuff. The hypothesis to invalidate is: maybe the types of people assembled into the groups we call "governments" are very unlikely to carry an "activate mutual destruction" decision all the way through. To be clear, I don't believe this, and I th... (read more)

How many times would nuclear weapons have been used if every state had them since 1950?

Great set of links, appreciate it. Was especially excited to see lukeprog's review and the author's presentation of Atomic Obsession.

I'm inclined toward answers of the form "seems like they would have been used more or some civilizational factor would need to change" (which is how I interpret Jackson's answer on strong global policing). Which is why I'm currently most interested in understanding the Atomic Obsession-style skeptical take.

If anyone is interested, the following are some of the author's claims which seem pertinent, at least as far as I can te... (read more)

Notes on 'Atomic Obsession' (2009)

Re direct military conflicts between nuclear weapons states: this might not exactly fit the definition of "direct" but I enjoyed skimming the mentions of nuclear weapons in this wikipedia on the yom kippur war, which saw a standoff between Israel (nuclear) and Egypt (not nuclear, but had reportedly been delivered warheads by USSR). There is some mention of Israel "threatening to go nuclear" possibly as a way of forcing the US to intervene with conventional military resources.

3Max_Daniel1y
Interesting, thank you! I hadn't been aware of this case.
How many times would nuclear weapons have been used if every state had them since 1950?

Interesting! For (1) how do you expect the economic superpowers to respond to smaller nations using nuclear weapons in this world? It sounds like because of MAD between the large nations, your model is that they must allow small nuclear conflicts, or alternatively pivot into your scenario 2 of increased global policing, is that correct?

1Jackson Wagner1y
Yes, that's what I'm thinking. As I'm continuing to develop this thought (sorry for being a bit repetitive in my posts), perhaps the main things that determine where the world might fall between scenarios (1) and (2) are: -How hard it is to establish stricter global governance: Is there an easy proliferation bottleneck that can be controlled, like ICBM technology or uranium mines? Can the leading nations get along well enough to cooperate on the shared goals of global governance? When everyone has nukes, how easy is it to boss around small countries? If the leading nations don't have the state capacity to pull off global governance, then we'll be stuck in a multipolar anything-goes world no matter what we think is preferable. -The "contagiousness" of nuclear conflict helps determine the value of strict global governance: if conflicts are extremely contagious (such that something like the real-world Syrian Civil War ends up with the superpowers at DEFCON 1), then small-scale wars are still extremely dangerous, and global policing is very desirable. If nuclear conflict isn't contagious at all and it's easy to stay out of a dispute, then it would be a lot more acceptable for the leading nations to just let nuclear wars happen, in the same way that the modern world often lets civil wars happen without intervening too much. Just play defense by being really paranoid about your ports/borders, and threatening to first-strike anyone who develops suspicious new long-range capabilities. I really don't know much about the question of contagiousness. Is there something special about nuclear weapons and the "nuclear taboo" that affects contagiousness? (Maybe nations feel like they have to "use or lose" their ICBMs before they are destroyed by opponents.) Or does all war seem contagious because it naturally erupts at the center of complex knots of geopolitical tensions and alliances? (Like the rapid domino-like declarations of war that set off WW1, or the agglomeration of seem
How likely is a nuclear exchange between the US and Russia?

Thanks for this post Luisa! Really nice resource and I wish I caught it earlier. A couple methodology questions:

1. Why do you choose an arithmetic mean for aggregating these estimates? It seems like there is an argument to be made that in this case we care about order-of-magnitude correctness, which would imply taking the average of the log probabilities. This is equivalent to the geometric mean (I believe) and is recommended for fermi estimates e.g. (here)[https://www.lesswrong.com/posts/PsEppdvgRisz5xAHG/fermi-estimates].

2. Do you have a sense for how

2Pablo1y
FYI, this post [https://forum.effectivealtruism.org/posts/sMjcjnnpoAQCcedL2/when-pooling-forecasts-use-the-geometric-mean-of-odds] by Jaime has an extended discussion of this issue.

Why do you choose an arithmetic mean for aggregating these estimates?

This is a good point.

I'd add that as a general rule when aggregating binary predictions one should default to the average log odds, perhaps with an extremization factor as described in (Satopää et al, 2014).

The reasons are a) empirically, it seems to work better, b) the way Bayes rules works it seems to suggest very strongly than log odds are the natural unit of evidence, c) apparently there are some complex theoretical reasons ("external bayesianism") why this is better (the ... (read more)

Make your own cost-effectiveness Fermi estimates for one-off problems

3Owen Cotton-Barratt1y
I'm not certain I remember what I was referring to here, but my best guess is that it was this article: https://forum.effectivealtruism.org/posts/HENbwrDYnTktRtNdE/report-allocating-risk-mitigation-across-time [https://forum.effectivealtruism.org/posts/HENbwrDYnTktRtNdE/report-allocating-risk-mitigation-across-time]
How to PhD

Strong +1 to this. I think I have observed people who have really good academic research taste but really bad EA research taste

How to PhD

Taste is huge! I was trying to roll this under my "Process" category, where taste manifests in choosing the right project, choosing the right approach, choosing how to sequence experiments, etc etc. Alas, not a lossless factorization

These exercises look quite neat, thanks for sharing!

How to PhD

Thanks Seb. I don't think I have energy to fully respond here, possibly I'll make a separate post to give this argument its full due.

One quick point relevant to Crux 2: "I can also think of many examples of groundbreaking basic science that looks defensive and gets published very well (e.g. again sequencing innovations, vaccine tech; or, for a recent example, several papers on biocontainment published in Nature and Science)."

I think there are many-fold differences in impact/dollar between the tech you build if you are trying to actually solve the problem a... (read more)

How to PhD

I bet it is! The example categories I think I had in mind at time of writing would be 1) people in ML academia who want to be doing safety instead doing work that almost entirely accelerates capabilities and 2) people who want to work on reducing biological risk instead publish on tech which is highly dual use or broadly accelerates biotechnology without deferentially accelerating safety technology.

I know this happens because I've done it. My most successful publication to date (https://www.nature.com/articles/s41592-019-0598-1) is pretty much entirely c... (read more)

How to PhD

This is interesting and also aligns with my experience depending on exactly what you mean!

• If you mean that it seems less difficult to get tenure in CS (thinking especially about deep learning) than the vibe I gave, (which is again speaking about the field I know, bioeng) I buy this strongly. My suspicion is that this is because relative to bioengineering, there is a bunch of competition for top research talent by industrial AI labs. It seems like even the profs who stay in academia also have joint appointment in companies, for the most part. There isn't
To clarify, I don't think tenure is guaranteed, more that there's significant margin of error. I can't find much good data on this, but this post [https://dynamicecology.wordpress.com/2014/07/21/dont-worry-too-much-about-whether-youll-get-tenure-because-you-probably-will/] surveys statistics gathered from a variety of different universities, and finds anywhere between 65% of candidates get tenure (Harvard) to 90% (Cal State, UBC). Informally, my impression is that top schools in CS are the higher end of this: I'd have guessed 80%. Given this, the median person in the role could divert some of their research agenda to less well-received topics and still get tenure. But I don't think they could work on something that no one in the department or elsewhere cared about. I've not noticed much tenure switch in CS but have never actually studied this, would love to see hard data here. I do think there's a significant difference in research agendas between junior and senior professors, but it's more a question of what was in vogue when they were in grad school and shaped their research agenda, than tenured vs non-tenured per se. I do think pre-tenure professors tend to put their students under more publication pressure though.
How to PhD

"Working backwards" type thinking is indeed a skill! I find it plausible a PhD is a good place to do this. I also think there might be other good ways to practice it, like for example seeking out the people who seem to be best at this and trying to work with them.

+1 on this same type of thinking being applicable to gathering resources. I don't see any structural differences between these domains.

How to PhD

This is an excellent comment, thanks Adam.

A couple impressions:

• Totally agree there are bad incentives lots of places
• I think figuring out what existing institutions have incentives that best serve your goals, and building a strategy around those incentives, is a key operation. My intent with this article was to illustrate some of that type of thinking within planning for gradschool. If I was writing a comparison between working in academia and other possible ways to do research I would definitely have flagged the many ways academic incentives are better

But compared to working with a funder who, like you, wants to solve the problem and make the world be good, any of the other institutions mentioned including academia look extremely misaligned.

I think overall the incentives set up by EA funders are somewhat better than run-of-the-mill academic incentives, but I think the difference is smaller than you seem to believe, and I think we're a long way from cracking it. I think this is something we can get better at, but ... (read more)

How much does performance differ between people?

Yeah this is great; I think Ed probably called them sleeping beauties and I was just misremembering :)

Thanks for the references!

How to PhD

Appreciate your comment! I probably won't be able to give my whole theory of change in a comment :P but if I were to say a silly version of it, it might look like: "Just do the thing"

So, what are the constituent parts of making scientific progress? Off the cuff, maybe something like:

1. You need to know what questions are worth asking / problems are worth solving
2. You need to know how to decompose these questions in sub-questions iteratively until a subset are answerable from the state of current knowledge
3. You need to have good research project management ski
How to PhD

Thanks Charles! I think of your two options I most closely mean (1). For evidence I don't mean 2: "Optimize almost exclusively for compelling publications; for some specific goals these will need to be high-impact publications."

My attempt to restate my position would be something like: "Academic incentives are very strong and its not obvious from the inside when they are influencing your actions. If you're not careful, they will make you do dumb things. To combat this, you should be very deliberate and proactive in defining what you want and how you want i... (read more)

Publishing good papers is not the problem, deluding yourself is.

Big +1 to this. Doing things you don't see as a priority but which other people are excited about is fine. You can view it as kind of a trade: you work on something the research community cares about, and the research community is more likely to listen on (and work on) things you care about in the future.

But to make a difference you do eventually need to work on things that you find impactful, so you don't want to pollute your own research taste by implicitly absorbing incentives or others opinions unquestioningly.

How to PhD

I am doing 1. 2 is an incidental from the perspective of this post, but is indeed something I believe (see my response to bhalperin). I think my attempt to properly flag my background beliefs may have led to the wrong impression here. Or alternatively my post doesn't cover very much on pursuing academia, when the expected post would have been almost entirely focused on this, thereby seeming like it was conveying a strong message?

In general I don't think about pursuing "sectors" but instead about trying to solve problems. Sometimes this involves trying to g... (read more)

2AllAmericanBreakfast1y
That makes sense. I like your approach of self-diagnosing what sort of resources you lack, then tailoring your PhD to optimize for them. One challenge with the "work backwards" approach is that it takes quite a bit of time to figure out what problems to solve and how to solve them. As I attempted this planning my own immanent journey into grad school, my views gained a lot of sophistication, and I expect they'll continue to shift as I learn more. So I view grad school partly as a way to pursue the ideas I think are important/good fits, but also as a way to refine those ideas and gain the experience/network/credentials to stay in the game. The "work backwards" approach is equally applicable to resource-gathering as finding concrete solutions to specific world problems. I think it's important for career builders to develop gears-level models of how a PhD or tenured academic career gives them resources + freedom to work on the world problems they care about; and also how it compares to other options. Often, people really don't seem to do that. They go by association: scientists solve important problems, and most of them seem to have PhDs and academic careers, so I guess I should do that too. But it may be very difficult to put the resources you get from these positions to use in order to solve important problems, without a gears-level model of how those scientists use those resources to do so.
How to PhD

Ugh. Shrug. That isn't supposed to be the point of this post. All my comments on this are to alert the reader that I happen to believe this and haven't tried to stop it from seeping into my writing. It felt disingenuous not to.

But since you raised, I feel like making it clear, if it isn't already, that I do not recommend reversing this advice. At least if you are considering cause areas/ academic domains that I might know about (see my preamble). I have no idea how applicable this is outside of longtermist technical-leaning work.

If you think you might be a... (read more)

I'm not convinced that academia is generally a bad place to do useful technical work. In the simplest case, you have the choice between working in academia, industry or a non-profit research org. All three have specific incentives and constraints (academia - fit to mainstream academic research taste; industry - commercial viability; non-profit research - funder fit, funding stability and hiring). Among these, academia seems uniquely well-suited to work on big problems with a long (10-20 year) time horizon, while having access to extensive expertise and col... (read more)

5antimonyanthony1y

You approximately can't get directly useful/ things done until you have tenure.

At least in CS, the vast majority of professors at top universities in tenure-track positions do get tenure. The hardest part is getting in. Of course all the junior professors I know work extremely hard, but I wouldn't characterize it as a publication rat race. This may not be true in other fields and outside the top universities.

The primary impediment to getting things done that I see is professors are also working as administrator and teaching, and that remains a problem post-tenure.

How much does performance differ between people?

Sorry meant to write "component of scientific achievement is predictable from intrinsic characteristics" in that first line

How much does performance differ between people?

Neat. I'd be curious if anyone has tried blinding the predictive algorithm to prestige: ie no past citation information or journal impact factors.  And instead strictly use paper content (sounds like a project for GPT-6).

It might be interesting also to think about how talent vs. prestige-based models explain the cases of scientists whose work was groundbreaking but did not garner attention at the time. I'm thinking, e.g. of someone like Kjell Keppe who basically described PCR, the foundational molbio method, a decade early.

If you look at natural  ... (read more)

How much does performance differ between people?

Interesting! Many great threads here. I definitely agree that some component of scientific achievement is predictable, and the IMO example is excellent evidence for this. Didn't mean to imply any sort of disagreement with the premise that talent matters; I was instead pointing at a component of the variance in outcomes which follows different rules.

Fwiw, my actual bet is that to become a top-of-field academic you need both talent AND to get very lucky with early career buzz. The latter is an instantiation of preferential attachment. I'd guess for each top-... (read more)

3Max_Daniel1y
No, they considered the full distribution of scientists with long careers and sustained publication activity (which themselves form the tail of the larger population of everyone with a PhD). That is, their analysis includes the right tail but wasn't exclusively focused on it. Since by its very nature there will only be few data points in the right tail, it won't have a lot of weight when fitting their model. So it could in principle be the case that if we looked only at the right tail specifically this would suggest a different model. It is certainly possible that early successes may play a larger causal role in the extreme right tail - we often find distributions that are mostly log-normal, but with a power-law tail, suggesting that the extreme tail may follow different dynamics.
1eca1y
Sorry meant to write "component of scientific achievement is predictable from intrinsic characteristics" in that first line
How much does performance differ between people?

Great post! Seems like the predictability questions is impt given how much power laws surface in discussion of EA stuff.

More precisely, future citations as well as awards (e.g. Nobel Prize) are predicted by past citations in a range of disciplines

I want to argue that things which look like predicting future citations from past citations are at least partially "uninteresting" in their predictability, in a certain important sense.

2Max_Daniel1y
A related phenomenon has been studied in the scientometrics literature under the label 'sleeping beauties' [https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=sleeping+beauties&btnG=] . Here is what Clauset et al. (2017, pp. 478f.) say in their review of the scientometrics/'science of science' field: [See doc linked in the OP for full reference.]
3Max_Daniel1y
Relatedly, you might be interested in these two footnotes discussing how impressive it is that Sinatra et al. (2016) - the main paper we discuss in the doc - can predict the evolution of the Hirsch index (a citation measure) over a full career based on the the Hirsch index after the 20 or 50 papers:

Thanks! I agree with a lot of this.

I think the case of citations / scientific success is a bit subtle:

• My guess is that the preferential attachment story applies most straightforwardly at the level of papers rather than scientists. E.g. I would expect that scientists who want to cite something on topic X will cite the most-cited paper on X rather than first looking for papers on X and then looking up the total citations of their authors.
• I think the Sinatra et al. (2016) findings which we discuss in our relevant section push at least slightly against a story
AMA: Holden Karnofsky @ EA Global: Reconnect

To operate in the broad range of cause areas openphil does, I imagine you need to regularly seek advice from external advisors. I have the impression that cultivating good sources of advice is a strong suite of both yours and OpenPhils.

I bet you also get approached by less senior folks asking for advice with some frequency.

As advisor and advisee: how can EAs be more effective at seeking and making use of good advice?

Possible subquestions: What common mistakes have you seen early career EAs make when soliciting advice, eg on career trajectory? When do you s... (read more)

Notes on "Bioterror and Biowarfare" (2006)

Interesting point. Note that a requirement for retaliation is knowledge of the actor to retaliate against. This is called “attribution” and is a historically hard problem for bioweapons which is maybe getting easier with modern ML (COI- I an a coauthor: https://www.nature.com/articles/s41467-020-19149-2)

Project Ideas in Biosecurity for EAs

Makes sense- possibly I'd change my mind about many of these after hearing the motivation. The second half of your response make me believe that we actually don't disagree that much RE a lot of the projects in here being good substantially or primarily because they could help establish a research track record or be a good learning opportunity.

Project Ideas in Biosecurity for EAs

Thanks for writing this David. Its been on my todo list for a while to write down project ideas like this. I think some of these ideas are useful and worth doing, and getting those out in the open is great.

On the other hand, I think its actually pretty hard to find research which is directly good for reducing biorisk. In my experience the space of ideas which “seem maybe useful” is much larger than the set of projects which actually directly help, on more reflection. This is a general problem and not intended to be a specific critique of the ideas you shar... (read more)

4Davidmanheim1y
I'd be interested in more specific private feedback on which projects you don't think would not be useful, or ideas for other things you think people with those skill sets could do that would be more useful. Cross-checking you intuitions with others would be good - for each of these projects, someone else working actively in biosecurity thought the project would be useful. And I think that it's easy to have a narrow view of what is useful - I wouldn't have thought people would want many of these answers until they explained that they did, and often why. That said, if someone is interested in working directly on things that are substantially important, and has a track record for doing so, there are people who want to hire them already, and they have plenty of opportunity for collaboration with biosecurity EAs. I didn't, and don't, think that we need to provide that set of people lists of things to work on - and I would agree that this isn't the set of highest priority tasks, many of which require funding and support, or have other reasons that people cannot pick them up as side projects. This list is geared towards things that people with a diverse skill set can do as an initial step, which active biosecurity researchers have said would show them someone is capable of doing useful research, while avoiding information hazards.
COVID-19 brief for friends and family

As anyone who have checked the google doc recently knows already, I haven't been maintaining it. It is now so out of data I consider it to be doing more harm then good, and have killed the link. I think most people have found better resources by now, anyway.