This is the second post out of two in a series about how the Forum gets made. In the last post, we introduced ourselves. In this post, we’ll try to lay out the models that guide our work and the plans we have for the future.

Models

The Forum is not a program executing a single plan; it’s a distributed ecosystem of many people trying to achieve many goals. The impacts are as numerous as the aims of every post and comment. Still, we can classify them using a few broad categories:

  • Sharing of existing ideas
  • Development and refinement of new ideas
  • Talent discovery
  • Public accountability
  • Spreading of norms
  • Encouraging coordination

(Credit to Oliver Habryka for an initial list upon which we built to produce the above.)

To achieve those goals (or really, a much broader class of goals), we think the Forum needs both of the following intermediate goals:

  • People reading good content
  • Good content for people to read (this includes both posts and comments)

We’ve developed a metric we use to figure out whether this is happening (and thus, whether we’re doing a good job). We evaluate our plans largely based on how we think they’ll impact the number of good posts [1] and the amount of views those posts receive. [2]

Don’t read too much confidence into this section. We view it as a better-than-nothing operating framework. In an ideal world, we’d have much more empirical data backing up our model. We’ve also had an actual, updating graph of the metric for all of a few weeks. Finally, we also track other metrics (from “time on site” to “mentions on popular blogs”) to get a better picture of the Forum’s impact.

Plans

I know the title of the section is “Plans,” but first, here are some things we’ve already done that we think of as aiming towards the “views on good posts” metric:

  • Built the timeframe feature into the All Posts page to enable easier post discovery
    • Since that page was hard to discover, we ported (and refactored) an improved sidebar from LW
  • Ported and modified the Community Favorites section and added it to the homepage
  • If you wanted, you could think of the Forum Prize as assisting with this metric, although it predates the theory
  • Posted an open offer for Forum editing from Aaron, which has substantially increased the number of people seeking feedback on posts, and hopefully the counterfactual quality of those posts (we had done this on an ad-hoc basis before the posting)

Quarterly goals for Q4

A lot of our goals aren’t explicit features. They’re things like “merge in LW with maximum 4 weeks delay” or “meet these goals every two-week sprint”. Our main relevant feature-goals are:

  • Creating a “Best posts since you last visited” section
    • Would allow users who visit infrequently to see the top-rated posts they haven’t been around for.
  • Porting the sequences feature
    • We’d build this feature to support the work on the EA Handbook 3.0. We probably won’t finish the Handbook this quarter, so this will remain hidden for a while.

There are other goals that aren’t large enough to quite be “quarterly goals” in the OKR system we use, but will likely be a sprint goal, such as fixing our (currently LessWrong-green) emails, adding an option to hide karma, and improving our error reporting.

The content side (aka Aaron), also has some goals pertaining to the Forum:

  • Encourage more people to do AMAs (like this one)
  • Get more blogs and orgs to crosspost their content
  • Write a few posts himself, and continue to encourage others to write up their ideas (e.g. from this thread)
  • Write / curate the first sections of a new version of the EA Handbook (though these may not appear on the Forum until they’ve been subject to feedback from a small group of advisors)

Later goals

We might do these, but not this quarter.

  • Technical SEO
    • We’re currently probably underperforming in Google search results relative to where we could be if we put effort into optimizing the site to match the whims of the great judge Google.
  • Category tagging
    • We definitely want to be cautious here, but we’re interested in investigating ways where people who are interested in a particular sub-field of EA can find the content they’re interested in more easily, and it’s possible that we could develop something around category tags to do this.
  • Recommendation algorithm
    • Currently the new Community Favorites section give you recommendations for posts to read that a) are popular, and b) you haven’t read. That’s not a very complicated algorithm. We could make it much smarter; think of the Netflix recommendation algorithm. I’d guess it could really increase the amount of good posts people read, but it would be a lot of work.

Final Thoughts

We hope, after reading this, you have a good sense of the actual dynamics that shape this Forum. Don’t be shy to ask questions! We’re especially interested in reactions to the proposed features.

Footnotes

[1] What’s a “good post”? Currently, we mark certain posts as representing the kind of content we most want on the Forum (this corresponds to roughly 30% of posts). We also include posts that sparked lively/productive conversations, even if the posts themselves were very brief/just asking a question. This doesn’t mean we don’t value lots of other content — we do! A lot! — but tracking views to all posts seems less informative than watching views on the posts we think offer the most value to those who read them. (This evaluation is cause-impartial and does not affect the visibility of posts in any way.)

[2] Currently, we’re focusing on views from logged-in users. This has several good properties:

  • We’re less tempted to Goodhart the metric by sharing our favorite posts as widely as possible, even if we don’t think it will be all that helpful to a broad audience
  • We can use our own database to measure views, rather than relying on Google Analytics (which works well, but doesn’t see users who have an ad-blocker turned on)
  • We get a better sense of how well posts are inspiring people to vote and comment, because every view we track is (in theory) someone who could do those things

24

3 comments, sorted by Highlighting new comments since Today at 3:35 PM
New Comment

I'd just like to +1 that SEO could be very valuable.

More generally, I'd like to see more work into solving the "lost content" problem. My guess is that great posts from last year are rarely read or remembered today while they might still be rather relevant. “Best posts since you last visited” could be helpful, but so would SEO and also maybe tags.

To my knowledge:

  • The CEA has two full-time developers (Sam Deere and JP Addison). JP has been at CEA since at least June 2018, and Sam has been there for many years.
  • The EA forum appears to be a copy of the LessWrong forum, with some minor cosmetic modifications (like having a 'community' section).
  • The two web development achievements mentioned in this post ('Built the timeframe feature into the All Posts page to enable easier post discovery', 'Ported and modified the Community Favorites section and added it to the homepage') are features directly ported from LessWrong.

Please correct me if I'm wrong about any of these.

My interpretation of these facts is that the developer team at CEA isn't very productive.

(There's also EA funds and the EA donor lotteries, but what new features have been added there in the last 12 months?)

Hi,

I'll first clarify / correct some facts, then discuss my interpretation.

  • I've been at CEA since September 2017. I have been working full time on the Forum since May. Previously, I was only trying to keep it up to date with LessWrong. As mentioned in the previous post in this series, Sam does not work directly on the Forum.
  • Correct. The developer term of art is "fork". In this case it's a fork where we push our changes upstream to LessWrong, and keep up to date with their changes.
  • When I say 'Built the timeframe feature' I do in fact mean that I built it. It exists on LessWrong because I submitted a Pull Request. For Community Favorites section I refactored a small amount of code to make it fit back into our frontpage, (as LessWrong's is used quite differently) and to have it include posts from the "Community" section at the user's option. These two features were selected for relevance to the metric, but don't represent the entirety of my work.

I’ll let Sam reply with the amount of behind-the-scenes work that he’s needed to do on the Funds & Giving What We Can (if he wants, also fair warning that he might want to wait till a workday).

As an open source project, we're quite a bit more open than the average software project, but I realize it’s hard to assess the work that goes into a tech project from the outside.

Given the state it was in previously and the commercial alternatives, I'm pretty happy with the progress made on the Forum.