All of Jsevillamol's Comments + Replies
I have received a private comment asking about the role of civil protection during COVID.
This is what I answered:
About COVID, we asked everyone about their role on it.
Basically the picture I got is: public health is delegated to the ministry of health, and in particular pandemics are seen as the business of the Centre of Sanitary Alerts (CCAES) The CCAES does have an early warning system, but we could not talk to them and we don't know how the systems reacted to COVID.
In the civil prot side, they basically did nothing until the govn declared an... (read more)
Echoing everyone else, thank you for all your hard work.
I do not exaggerate when I say you are the best forum moderator I have ever seen. I am really impressed with your availability, creativity and kindness. You have driven the culture of this website to a whole new level, and inspired me and I bet many others to write better content.
Good luck at OpenPhil!
I think that this kind of criticism is really useful, and I am glad it was written.
That being said, there is something about this post that really rubbed me the wrong way. This is a shame, because the topic is very pertinent and deserves an in-depth discussion - how do opportunities for funding today compare to opportunities for funding 5 or 10 years from now?
Let me try to give my best shot at a more thoughtful critique.
... (read more)In the midst of a global pandemic that pushed 150 million people into extreme poverty while billionaires’ wealth grew by
"I assume that the crux here is that GiveDirectly believes that spending more money now would have a good publicity effect, that would promote philanthropy and raise the total amount of donations overall.
I would change my mind if this was the case, but I don't see this as obvious."
I'm not entirely sure what the answer is here either, but one thought I had today was "I should make a Facebook post for Thanksgiving/Christmas telling my friends why I think it's so important to donate to GiveWell - your marginal donation can save a life for $3-5k! Ah, but... (read more)
I agree with most of your comment.
However, given that GiveWell want to use a bar of 5-7x GiveDirectly, I think accounting for a study that at best will demonstrate that GiveDirectly is 2.6 times more effective than previously thought, will not influence GiveWell’s decision to wait for better opportunities, since it still doesn’t meet the 5-7x GiveDirectly bar.
I do think so!
It's hard to contest that change across many dimensions has been accelerating.
And it would make sense that this accelerating change makes parental advice less applicable, and thus parents less influential overall.
The A,B,C example you came up with is certainly a strike against average log odds and in favor of average probs.
I have though more about this. I now believe that this invariance property is not reasonable - aggregating outcomes is (surprisingly) not a natural operation in Bayesian reasoning. So I do not think this is a strike agains log-odd pooling.
Why did you end up being turned off by EA?
It's hard to pinpoint but I think it's somehting along the lines of a) the messaging didn't match my perceived self-image ("I am not an altruist"), b) they seemed weirdly fanatical ("donating 10% of my money seems crazy weird") and c) I was not impressed with the people I interacted with (concretely the people from eg the rationality community seemed comparatively more thoughful and to be working on cooler things).
I am unsure of whether I would have changed my mind had I interacted more with the community at ... (read more)
Retreats are awesome!
It was the MIRI Summer Fellows in 2015. For full disclosure it was not about EA, and I came off it being turned off by EA aesthetics. But it was where I first heard about the movement, and it was crucial for my involvement in the long term.
One hour (maybe two) fellowship sessions isn’t long enough to get into “late night life-changing conversations” mode, which is important for big changes.
This to me is the main downside.
I got introduced to EA over a 3 week in-person summer program, and my experience is that 2~4 week in person intensive programs have a good track record in getting people excited and engaged. Off the top of my head 1 out of 3 participants in the camps ive been involved in became counterfactually engaged, 1 out of three was engaged but would be anyway and 1 out of ... (read more)
Thank you!
These papers were ones that William MacAskill was considering citing in his forthcoming book. FF hired me to thoroughly check them.
There is definitely many other persistence papers I didn't cover!
Eg:
- Acemoglu et al, Colonial Origins
- Acemoglu et al, Reversal of Fortune
- Woodberry (2012). The Missionary Roots of Liberal Democracy
- All the papers cited in Kelly's Understanding Persistence
And many others.
EDIT: Faatima Osman questioned whether it was fair to exclude respondents from Benin, Ghana and Nigeria in Nunn and Wantchekon's paper, given that Nigeria is the most populated country in Africa by far.
And in hindsight I think she is totally right - respondents from these countries are ~25% of the sample! I now believe that its unfair to call these respondents outliers. Correspondingly, my trust in Nunn and Wantchekon's paper has gone up, since Kelly's critique was my main concern about it.
EDIT: Faatima Osman questioned whether it was fair to exclude respondents from Benin, Ghana and Nigeria in Nunn and Wantchekon's paper, given that Nigeria is the most populated country in Africa by far.
And in hindsight I think she is totally right - respondents from these countries are ~25% of the sample! I now believe that its unfair to call these respondents outliers. Correspondingly, my trust in Nunn and Wantchekon's paper has gone up, since Kelly's critique was my main concern about it.
Thank you for your thoughful reply. I think you raise interesting points, which move my confidence in my conclusions down.
Here are some comments
[...] averaging log odds will always give more extreme pooled probabilities than averaging probabilities does
As in your post, averaging the probs effectively erases the information from extreme individual probabilities, so I think you will agree that averaging log odds is not merely a more extreme version of averaging probs.
I nonetheless think this is a very important issue - the difficulty of separating the extrem... (read more)
On getting research collaborators
(adapted from a private conversation)
The 80/20 advice I would give is: be proactive in reaching out to other people and suggesting to them to work for an evening on a small project, like writing a post. Afterwards you both can decide if you are excited enough to work together on something bigger, like a paper.
For more in depth advice, here are some ways I've started collaborations in the past:
- Deconfusion sessions.
I often invite other researchers for short sessions of 1-2 hours to focus on a topic, with the goal of cod
Thank you! I am quite honoured. And congratulations to the other winners!
I absolutely love the work you have done, thank you so much!
He is listed in the website.
> OpenAI is governed by the board of OpenAI Nonprofit, which consists of OpenAI LP employees Greg Brockman (Chairman & CTO), Ilya Sutskever (Chief Scientist), and Sam Altman (CEO), and non-employees Adam D’Angelo, Holden Karnofsky, Reid Hoffman, Shivon Zilis, Tasha McCauley, and Will Hurd.
It might not be up to date though
It can’t be up to date, since they recently announced that Helen Toner joined the board, and she’s not listed.
Note that Eliezer Yudkowski argument in the opening link is that OpenAI's damage was done by fragmenting the AI Safety community on its launch.
This damage is done - and I am not sure it bears much relation to what OpenAI is trying to do going forward.
(I am not sure I agree with Eliezer on this one, but I lack details to tell if OpenAI's launch really was net negative)
I found some revelant discussion in the EA Forum about extremizing in footnote 5 of this post.
... (read more)The aggregation algorithm was elitist, meaning that it weighted more heavily forecasters with good track-records who had updated their forecasts more often. In these slides, Tetlock describes the elitism differently: He says it gives weight to higher-IQ, more open-minded forecasters. The extremizing step pushes the aggregated judgment closer to 1 or 0, to make it more confident. The degree to which they extremize depends on how diverse and sophisticated the pool o
I liked this one a lot!
It was very easy to read and pulled me in. I felt compelled by the protagonists inner turmoil, and how he makes his decision. The writing was clear but it flowed very well. This is something I will send some friends to introduce them to effective altruism.
The only part I didn't like was the ending. I like the intention of linking to GiveWell's page but it pulled me totally out of the fantasy. Also the friend felt a bit 2D. But these are minor quibbles.
Thank you for writing this!
Thanks for this post - I think this was a very useful conversation to have started (at least for my own work!), even if I'm less confident than you in some of these conclusions
Thank you for your kind words! To dismiss any impression of confidence, this represents my best guesses. I am also quite confused.
I've heard other people give good-sounding arguments for other conclusions
I'd be really curious if you can dig these up!
... (read more)You later imply that you think [the geo mean of probs outperforming the geo mean of odds] is at least partly because of a specific
I get what you are saying, and I also harbor doubts about whether extremization is just pure hindsight bias or if there is something else to it.
Overall I still think its probably justified in cases like Metaculus to extremize based on the extremization factor that would optimize the last 100 resolved questions, and I would expect the extremized geo mean with such a factor to outperform the unextremized geo mean in the next 100 binary questions to resolve (if pressed to put a number on it maybe ~70% confidence without thinking too much).
My reasoning here is... (read more)
Hmm good question.
For a quick foray into this we can see what would happen if we use our estimate the mean of the max likelihood beta distribution implied by the sample of forecasts .
The log-likelihood to maximize is then
The wikipedia article on the Beta distribution discusses this maximization problem in depth, pointing out that albeit no closed form exists if and can be assumed to be not too small the max likelihood estimate can be approximated as ... (read more)
I was curious about why the extremized geo mean of odds didn't seem to beat other methods. Eric Neyman suggested trying a smaller extremization factor, so I did that.
I tried an extremizing factor of 1.5, and reused your script to score the performance on recent binary questions. The result is that the extremized prediction comes on top.
This has restored my faith on extremization. On hindsight, recommending a fixed extremization factor was silly, since the correct extremization factor is going to depend on the predictors being aggregated and the... (read more)
I don't think I get your argument for why the approximation should not depend on the downstream task. Could you elaborate?
Your best approximation of the summary distribution is already "as good as it can get". You think we should be cautious and treat this probability as if it could be higher for precautionary reasons? Then I argue that you should treat it as higher, regardless of how you arrived at the estimate.
In the end this circles back to basic Bayesian / Utility theory - in the idealized framework your cr... (read more)
I think this is a good account of the institutional failure example, thank you!
I think it was an intentional false alarm, to better simulate Petrov's situation
Similarly, I would like for comments I have minimised to stay minimised between visits (unless there is a new reply in thread)
The ideal system would [not] aggregate first into a single number [...] Instead, the ideal system would use the whole distribution of estimates
I have been thinking a bit more about this.
And I have concluded that the ideal aggregation procedure should compress all the information into a single prediction - our best guess for the actual distribution of the event.
Concretely, I think that in an idealized framework we should be treating the expert predictions as Bayesian evidence for the actual distribution of the event of interest&nb... (read more)
I've been having some mixed feelings about some recent initiatives in the Forum.
These include things in the space of the creative fiction contest, posting humorous top level content and asking people to share memes.
I am having trouble articulating exactly what is causing my uneasiness. I think its something along the lines of "I use the EA Forum to stay up to date on research, projects and considerations about Effective Altruism. Fun content distracts from that experience, and makes it harder for the work I publish in the Forum to be taken seriously"... (read more)
Some discussion about profile pictures for the Forum here
Done, thank you!
Here is something I wrote, and received some positive feeback on: Standing on a pile of corpses
Thank you! I learned too from the examples.
One question:
In particular, that the best approach for practical rationality involves calculating things out according to each of the probabilities and then aggregating from there (or something like that), rather than aggregating first.
I am confused about this part. I think I said exactly the opposite? You need to aggregate first, then calculate whatever you are interested in. Otherwise you lose information (because eg taking the expected value of the individual predictions loses information that was contain... (read more)
Thank you for your thoughts!
I agree with the general point of "different situations will require different approaches".
From that common ground, I am interested in seeing whether we can tease out when it is appropriate to use one method against the other.
*disclaimer: low confidence from here onwards
I do not find the first example about value 0 vs value 500 entirely persuasive, though I see where you are coming from, and I think I can see when it might work.
The arithmetic mean of probabilities is entirely justified when aggregating predictions from &nb... (read more)
Let's work this example through together! (but I will change the quantities to 10 and 20 for numerical stability reasons)
One thing we need to be careful with is not mixing the implied beliefs with the object level claims.
In this case, person A's claim that the value is is more accurately a claim that the beliefs of person A can be summed up as some distribution over the positive numbers, eg a log normal with parameters and . So the density distribution of beliefs of A is ... (read more)
I think you mean bearish
Oops yes 🐻
You point out this highly skilled management/leadership/labor is not fungible
Yes, exactly.
I think what I am pointing towards is something like "if you are one such highly skilled editor, and your plan is to work on something like this part time delegating work to more junior people, then you are going to find yourself burnt out very soon. Managing a team of junior people / people who do not share your aesthetic sense to do highly skilled labor will be, at least for the first six months or so, much more work than if... (read more)
I am more bullish about this. I think for distill to succeed it needs to have at least two full time editors committed to the mission.
Managing people is hard. Managing people, training them and making sure the vision of the project is preserved is insanely hard - a full time job for at least two people.
Plus the part Distill was bottlenecked on is very high skilled labour, which needed a special aesthetic sensitivity and commitment.
50 senior hours per draft sounds insane - but I do believe the Distill staff when they say it is needed.
This wraps back to why ... (read more)
Create a journal of AI safety, and get prestigious people like Russell publishing on them.
Basically many people in academia are stuck chasing publications. Aligning that incentive seems important.
The problem is that journals are hard work, and require a very specific profile to push it forward.
Here is a post mortem of a previous attempt: https://distill.pub/2021/distill-hiatus/
From your comment, I just learned that Distill.pub is shutting down and this is sad.
The site was beautiful. The attention to detail, and attention to the reader and presentation were amazing.
Their mission seems relevant to AI safety and risk.
Relevant to the main post and the comment above, the issues with Distill.pub seem not to be structural/institutional/academic/social—but operational, related to resources and burnout.
This seems entirely fixable by money, maybe even a reasonable amount compared to other major interventions in the AI/longtermist sp... (read more)
META: Do you think you could edit this comment to include...
- The number of questions, and aggregated predictions per question?
- The information on extremized geometric mean you computed below (I think it is not receiving as much attention due to being buried in the replies)?
- Possibly a code snippet to reproduce the results?
Thanks in advance!
import requests, json
import numpy as np
import pandas as pd
def fetch_results_data():
response = {"next":"https://www.metaculus.com/api2/questions/?limit=100&status=resolved"}
results = []
while response["next"] is not None:
print(response["next"])
response = json.loads(requests.get(response["next"]).text)
results.append(response["results"])
return sum(results,[])
all_results = fetch_results_data()
binary_qns = [q for q in all_results if q['possibilities']['type'] == 'binary' and q['resolution'] in [0,1]]
biYou are right and I should be more mindful of this.
I have reformulated the main equations using only commonly known symbols, moved the equations that were not critical for the text to a footnote and added plain language explanations to the rest.
(I hope it is okay that I stole your explanation of the geometric mean!)
I mean in the past people were underconfident (so extremizing would make their predictions better). Since then they've stopped being underconfident. My assumption is that this is because the average predictor is now more skilled or because more predictors improves the quality of the average.
Gotcha!
The bias isn't that more questions resolve positively than users expect.
Oh I see!
but also the average predictor improving their ability also fixed that underconfidence
What do mean by this?
Metaculus has a known bias towards questions resolving positive
Oh I see!
It is very cool that this works.
One thing that confuses me - when you take the geometric mean of probabilities you end up with . So the pooled probability gets slighly nudged towards 0 in comparison to what you would get with the geometric mean of odds. Doesn't that mean that it should be less accurate, given the bias towards questions resolving ... (read more)
(I note these scores are very different than in the first table; I assume these were meant to be the Brier scores instead?)
Thank you for the superb analysis!
This increases my confidence in the geo mean of the odds, and decreases my confidence in the extremization bit.
I find it very interesting that the extremized version was consistently below by a narrow margin. I wonder if this means that there is a subset of questions where it works well, and another where it underperforms.
One question / nitpick: what do you mean by geometric mean of the probabilities? If you just take the geometric mean of probabilities then you do not get a valid probability - the sum of the pooled ps and... (read more)
Thank you!
The freedom for side projects is the best - though I should warn other people here than having a supportive supervisor who is okay with this is crucial.
I have definitely heard more than one horror story from colleagues who were constantly fighting their supervisors on the direction of their research, and felt they had little room for side projects.
EAF helped pass a ballot which doubled Zurich's development aid
Definitely one of my favorite examples, and one we're using now:
Still, that's exactly what makes this a good suggestion! If I'd forgotten to add the initiative, this would have been a critical reminder.
But when I look at this chart, my main takeaway is that there’s a ton of money being spent on welfare, and that working to make sure that money is spent as efficiently as possible could have a huge impact.
I think this is basically true.
A while back I thought that it was false - in particular, I thought that the public money was extremely tight, and that fighting to change a budget was an extremely political issue where one would face a lot of competition.
My experience collaborating with public organizations and hearing from public servants so far has be... (read more)
Reading time is one of my favourite features from Medium, and has helped me understand which of my posts are most useful. This in turn has informed my decisions on what to focus on writing.
I expect to get similar benefits if a feature like this is implemented in the EA Forum.
... (read more)In Section 4 we shift attention to the computational complexity of agreement, the subject of our deepest technical result. What we want to show is that, even if two agents are computationally bounded, after a conversation of reasonable length they can still probably approximately agree about the expectation of a [0, 1] random variable. A large part of the problem is to say what this even means. After all, if the agents both ignored their evidence and estimated (say) 1/2, then they would agree before exchanging even a single message. So agreement is only in
(disclaimer: this is my opinion)
In short: Spanish civil protection would not as of today consider making plans to address specific GCRs
There is this weird tension where they believe that resilience is very important, and that planning in advance is nearly useless for non-recurring risks.
The civil protection system is very geared towards response. Foresight, mitigation and prevention seldom happens.This means they are quite keen on improving their general response capacity but they have no patience for hypotheticals. So they would not consider specifi... (read more)