Hey Vasco —

Thanks for your interest and also for raising this with us before you posted so I could post this response quickly!

I think you are asking about the first of these, but I'm going to include a few notes on the 2nd and 3rd too as well just in case, as there's a way of hearing your question as about them. 

1. What is the internal process by which these rankings are produced and where do you describe it? 
2. What are problems and paths being ranked by? What does the ranking mean?
3. Where is our reasoning for why we rank each problem or path the way we do? 

We've written some about these things on our site. We’re on the lookout for ways to improve our processes and how we communicate about them (e.g. I updated our research principles and process page this year and would be happy to add more info if it seemed important. If some of the additional notes below seem like they should be included that'd be helpful to hear.) 

Here's a summary of what we say now with some additional notes:

On (1):

Our "Research principles and process" page is the best place to look for an overview, but it doesn't describe everything. 

I'll quote a few relevant bits here:

> Though most of our articles have a primary author, they are always reviewed by other members of the team before publication.

> For major research, we send drafts to several external researchers and people with experience in the area for feedback.

> We seek to proactively gather feedback on our most central positions — in particular, our views on the most pressing global problems and the career paths that have the highest potential for impact, via regularly surveying domain experts and generalist advisors who share our values.

> For some important questions, we assign a point person to gather input from inside and outside 80,000 Hours and determine our institutional position. For example, we do this with our list of the world’s most pressing problems, our page on the top most promising career paths, and some controversial topics, like whether to work at an AI lab. Ultimately, there is no formula for how to combine this input, so we make judgement calls [...] Final editorial calls on what goes on the website lie with our website director. [me, Arden]

> Finally, many of our articles are authored by outside experts. We still always review the articles ourselves to try to spot errors and ensure we buy the arguments being made by the author, but we defer to the author on the research (though we may update the article substantively later to keep it current).

Here are some additional details that aren't on the page:

To reply to your specific question about aggregating people's personal rankings: no, we don't do any formal sort of 'voting' system like that. The problems and paths rankings are informed by the views of the staff at 80,000 Hours and external advisors via surveys where I elicit people's personal rankings, and lots of ongoing internal discussion, but I am the "point person" for ultimately deciding how to combine this information into a ranking. In practice, this means my views can be expected to have an outsized influence, but I put a lot of emphasis on takes from others and aim for the lists to be something 80,000 Hours as an organisation can stand behind. Another big factor is what the lists were before, which I tend to view as a prior to update from, and which were informed by the research we did in the past and the views of people like like Ben Todd, Howie Lempel, and Rob Wiblin.

Our process has evolved over the years, and, for example, the formal "point person" system described above is recent as of this year (though it was informally something a bit like that before). I expect it'll continue to change, and hopefully improve, especially as we grow the team (right now we have only 2 research staff).

Sometimes it's been a while since we've looked at a problem or path, and we decide to re-do the article on it. That might trigger a change in ranking if we discover something that changes our minds.

More often we adjust the rankings over time without necessarily first re-doing the articles, often in response to surveys of advisors and team members, feedback we get, or events in the world. This might then trigger looking more into something and adding or re-doing a relevant article. 

The rankings are not nearly as formal or quantitative as, e.g. the cost-effectiveness analyses that GiveWell performs of its top charities. Though previous versions of the site have included numerical weightings to something like the problem profiles list, we’ve moved away from that practice. We didn’t think the BOTECs and estimations that generated these kinds of numbers were actually driving our views, and the numbers they produced seemed like they suggested a misleading sense of precision. Ranking problems and career paths is messy and we aren't able to be precise. We discuss our level of certainty in e.g. the problem profiles FAQ and at the end of the reserach principles page and try to reflect it in the language on the problems and career path pages. 

As you noted, when we make a big change, like adding a new career path to the priority paths, we try to announce it in some prominent form, though we don't always end up thinking it's worth it. E.g. we sent a newsletter in April explaining why we now consider infosec to be a priority path. We made a similar announcement when we added AI hardware expertise to the priority paths. Our process for this isn't very systematic.

On (2): 

For problems: In EA shorthand, the ranking is via the ITN framework. We try to describe that in a more accessible / short way at the top of the page in the passage you quoted.

We also have an FAQ which talks a bit more about it.

For career paths it is slightly more complicated. A factor we weren't able to fit into the passage you quoted is: we also down-rank paths if they are super narrow/most people can't follow them (or don't write about them at all) – e.g. becoming a public intellectual (or to take an extreme example, becoming president of the US.)

On (3):

For the most part, we want the articles themselves to explain our reasoning – in each problem profile or career review, we say why we think it's as pressing / promising as we think it is. 

We also draw on surveys of 80k staff + external advisors to additionally help determine and adjust the ranking over time, as described above. We don't publish these surveys, but we describe the general type of person we tend to ask for input here.

Best,
Arden

I, too, would be happy to see more transparency about the 80.000 hour rankings. I think it would be especially valuable to see to which degree they reflect the individual judgment of decision-makers. I would also be interested in whether they take into account recent discussions/criticisms of model choices in longtermist math that strike me as especially important for the kind of advising 80.000 hours does (tldr: I take one crux of that article to be that longtermist benefits by individual action are often overstated, because the great benefits longtermism advertises require both reducing risk and keeping overall risk down long-term, which plausibly exceeds the scope of a career/life). 

I think this would help me with a more general worry I have, and maybe others share. As a teacher at a university, I often try to encourage students to rethink their career choices from an EA angle. 80.000 hours is a natural place to recommend for interested students, but I am wary of recommending it to non-longtermist students. Probably good seems to offer a more shorttermist alternative, but are significantly newer and have less brand recognition. I think there would be considerable value in having the biggest career-advising organization (80k) be a non-partisan EA advising organization, whereas I currently take them to be strongly favoring longtermism in their advice. While I feel this explicit stance is a mistake, I feel like getting a better grasp on its motivation would help me understand why it was taken.

I may be mistaken in taking 80.000 hours to lean heavily longtermist and Probably Good leaning heavily shorttermist, and would be happy to be corrected!