Some past example that come to mind. Kudos to all of the people mentioned for trying ambitious things, and writing up the retrospectives:

1. Not strictly speaking "EA", but an early effort from folks in the rationality community started an evidence-based medicine organization called MetaMed

Zvi Mowshowitz's post-mortem: https://thezvi.wordpress.com/2015/06/30/the-thing-and-the-symbolic-representation-of-the-thing/

Sarah Constantin's post-mortem: https://docs.google.com/document/d/1HzZd3jsG9YMU4DqHc62mMqKWtRer_KqFpiaeN-Q1rlI/edit

2. Michael Plant has a post-mortem of his mental health app, Hippo

3. Looking at around, I also found this list

Some other posts are the Good Technology Projects' postmortem, a postmortem of a mental health app by Michael Plant, organisations discuss their learnings in retrospectives like Fish Welfare Initiative or in posts announcing decisions to shut down like Students for High Impact Charities. In the Rationalist community, there was the Arbital Postmortem. You can see more examples on the Forum postmortems and retrospectives tag, and examples from the LessWrong community in their analogous tag.

I have failed to do any meaningful work on recommender systems alignment. We launched an association, YouTube acknowledged the problem with disinformation when we talked to them privately (about COVID, for example, coming from Russia, for example), but said they will not do anything, with or without us. We worked alone, I was the single developer. I burned out to the point of being angry and alienating people around me (I understand what Timnit Gebru has went through, because Russia (my home country) is an aggressor country, and there is a war in Tigray as well, which is related to Ethiopia, her home country). I have sent many angry/confusing emails that made perfect sense for me at the time... I went through homelessness and unemployment after internships at CHAI Berkeley and Google and a degree from a prestigious European university. I felt really bad for not being able to explain the importance of the problem and stop Putin before it was too late... Our colleagues' papers on the topic were silenced by their employers. Now I'm slowly recovering and feel I want to write about all that, some sort of a guide / personal experience on aligning real systems / organizations, and that real change comes really, really hard.

I've failed a few times. My social instincts tried to get me not to post this comment, in case it makes it more likely that I fail again, and failing hurts. I suspect there's really strong survivorship bias here.

In 2017 I quit my job and spent a significant amount of time self studying ML, roughly following a curriculum that Dario Amodei laid out in an 80k podcast. I ran this plan past a few different people, including in an 80k career advising session, but after a year, I didn't get a job offer from any of the AI Safety orgs I'd applied to (Ought, OpenAI, maybe a couple of others) and was quite burned out and demotivated. I didn't even feel up to trying to interview for an ML focused job. Instead I went back to web development (although it was with a startup that did suggest I'd be able to do some ML work eventually, but that job ultimately wasn't a great fit, and I moved on to my current role... as a senior web dev.) 

I think there are a bunch of lessons I learned from this exercise, but overall I consider it one of my failures. 

And presumably, if no one has failed, then people aren't trying things that are ambitious enough. Kat woods I think can speak to some incubations from Charity Entrepreneurship that didn't take off.

I can say that I failed at what I would consider a high risk, high reward project. I was a member of a charity entrepreneurship cohort and worked on an a nonprofit idea focused on advocacy for a pigouvian tax but unfortunately couldn't really get things off the ground for a few reasons. That said, I still highly recommend trying something ambitious. That failure taught me a lot and got me more into the policy realm which helped pave the way for my current work doing policy advisory in Congress which I think is relatively high impact. 

I think some of the worst failures are mediocre projects that go sort-of okay and therefore continue to eat up talent for a much longer time than needed; cases where ambitious projects fail to "fail fast". It takes a lot of judgment ability and self-honesty to tell that it's a failure relative to what one could have worked on otherwise.

One example is Raising for Effective Giving, a poker fundraising project that I helped found and run. It showed a lot of promise in terms of $ raised per $ spent over the years it was operating, and actually raised $25m for EA charities. But it looks a lot less high-impact once you draw comparisons to GWWC and Longview, or once you account for the small market size of the poker industry, lack of scalability, the expected future funding inflows into EA, and compensation from top Earning To Give opportunities. $25 million is really not much compared to the billions others raised through billionaire fundraising and entrepreneurship.

I personally failed to admit to myself that the project was showing mediocre but not amazing results, and only my successor (Stefan) then discontinued the project, which in hindsight seems like the correct judgment call.

I think it would be great to have some directory of attempted but failed projects. Often I've thought "Oh I think X is a cool idea, but I bet someone more qualified has already tried it, and if it doesn't exist publicly then it must have failed" but I don't think this is often true (also see this shortform about the failure of the efficient market hypothesis for EA projects). Having a list of attempted but shut down (for whatever reason) projects might encourage people to start more projects, as we can really see how little of the idea space has been explored in practice.

There's a few helpful write-ups (e.g. shutting down the longtermist incubator) but in addition to detailed post-mortems, I would be keen to see a low-effort directory (AirTable or even Google Sheets?) of attempted projects, who tried, contact details (with permission), why it stopped, etc. If people are interested in this, I can make some preliminary spreadsheet that we can start populating, but other recommendations are of course welcome.

I think it is fair to call EA Ventures, the Pareto Fellowship, and EA Grants failed projects. I’ve written evaluations of each of them here. I also discuss the EA Librarian project (though I don’t conduct a full evaluation) which also seems to have failed. My post also discusses numerous instances where EA projects that didn't fail still had significant and repeated problems. As such, I think it's relevant to your questions of: "what are concrete examples of EAs or EA orgs not meeting high expectations despite trying really hard? Is it possible that we just underestimate how successful someone with an EA mindset and the right support can be when they try really hard?"

What we learned from a year incubating longtermist entrepreneurship

Beyond asking about projects in a vague, general sense, it could also be interesting to compare the probabilities of success grantmakers in EA assign to their grantees' projects, to the fraction of them that actually succeed.