As with any social movement, people disagree about the best ways to take action. There are many critiques of EA which you should read to get a better idea of where others are coming from, for example, this post about effective altruism being an ideology, this post about someone leaving EA, this post about EA being inaccessible, or this post about blindspots in EA/rationalism communities. 

Even before SBF, many people had legitimate issues with EA from a variety of standpoints. Some people find the culture unwelcoming (eg. too elitist/not enough diversity), some people take issue with longtermism (eg. too much uncertainty), others disagree with consequentialism/utilitarianism, and still others are generally on board but find more specific issues in the way that EA approaches things. 

Post-SBF it's difficult to say what the full effects will be, but I think it's fair to say that SBF represents what many people fear/dislike about EA (eg. elitism, inexperience, ends-justifies-the-means reasoning, tech-bro vibes, etc). I'm not saying these things are necessarily true, but most people won't spend hundreds of hours engaging with EA to find out for themselves. Instead, they'll read an article on the New York Times about how SBF committed fraud and is heavily linked to EA and walk away with a somewhat negative impression. That isn't always fair, but it also happens to other social movements like feminism, Black Lives Matter, veganism, environmentalism, etc. EA is no exception, and FTX/SBF was a big enough deal that a lot of people will choose not to engage with EA going forward. 

Should you care? I think to an extent, yes - you should engage with criticisms, think through your own perspective, decide where you agree/disagree, and work on improving things where you think they should be improved going forward. We should all do this. Ignoring criticisms is akin to putting your fingers in your ears and refusing to listen, which isn't a particularly rational approach. Many critics of EA will have meaningful things to say about it and if we truly want to figure out the best ways to improve the world, we need to be willing to change (see: scout mindset). That being said, not all criticisms will be useful or meaningful, and we shouldn't get so caught up in the criticism that we stop standing for something. 

It is just people trying to figure out the best way to improve the world. What could be controversial about that?

How to best improve the world is far from straightforward, and EA hardly has a bulletproof or incontrovertible position on what needs to be improved or how. Even if you can get broad agreement on something like "we should fix poverty", there are dozens of questions about why poverty happens, how the root causes should or shouldn't be approved, what is the most effective and most ethical ways of addressing it (which are not necessarily always the same), etc.   I think the comments from S. E. Montgomery and bruce do an excellent job summing up  a lot of the points where EA is open to (often well-deserved) critiques.

Even before the whole FTX thing, EAs were being vilified on social media and even in academia.

I think there's some value in separating  the "public" image and criticism of EA from academic ones, and I wanted to comment on the academic aspect in particular.  While I'm sure that, like all movements, EA has been subject to vilification by academics, the academically published critiques of EA I've read tend to be fair, reasoned critiques primarily stemming from different approaches to doing good: 

• Institutional critiques that argue that EA is too focused on "band-aid" solutions that don't address the root causes of the problems it addresses, which would require institutional or radical change.
• Methodological critiques: this includes critiques of utilitarianism as a method for cause and intervention prioritization in general, and specific criticisms of calculations that address biases or models.
•  More general critiques on what should be prioritized for doing good: utilitarian or deontological views
• Critiques that argue the value of local (read: community based, bottom up) vs. global (read: top-down foreign aid and intervention) approaches
•  Critiques of the charity in general along the lines of the nonprofit industrial complex that are extended to EA.
• Critiques of charity in general as a means of redistribution of wealth and prioritizing who gets access to which services as opposed to democratic methods. 
• Anti-capitalist critiques that argue that EA is too tied in with capitalist ideals and methods to affect real change when it comes to poverty, which is considered to be a symptom of capitalism.

I don't think that all these critiques are excellent or as well backed up as they could be, but I think it's important to recognize that there are good reasons to find EA objectionable, controversial or even, on net, harmful based on any of these without having cognitive dissonance, resentment, or psychological effects involved. 

I think it’s badly named!

The name “effective altruism” has a holier than thou, “we’ve solved ethics and are perfectly ethical” vibe, which means that’s people are extra critical of EA when they disagree with the EA community or feel that the EA community isn’t doing the most good

Why wouldn't it be controversial? It suggests something other than people acting according to their personal pet projects, ideologies, and social affiliations, and proposes a way by which those can be compared and found wanting. The fact that it also comes with significantly more demandingness than anything else just makes it a stronger implicit attack.

Most people will read EA as a claim to the moral high ground, regardless of how nicely it's presented to them. Largely because it basically is one. Implicit in all claims to the moral high ground - even if it's never stated and even if it's directly denied - is the corollary claim that their claims to the moral high ground are lesser or even invalid. Which is a claim of superiority.

That will produce defensiveness and hostility by default.

Many people's livelihoods depend on ineffective charity, of course, and Sinclair's Rule is also a factor. But it's a minor one. The main factor is that the premise of EA is that charity should be purchasing utilons. And  even starting to consider that premise makes a lot of people tacitly realize that their political and charitable work may have been purchasing warm fuzzies, which is an unpleasant feeling that they are motivated to push back against to protect their self-image as a do-gooder/good person/etc.

Of course, there is no need for contradiction. You can purchase both utilons and warm fuzzies, so long as you do it separately. But in my estimation, no more than 5% of the world, at the absolute most, is amenable to buying warm fuzzies and utilons separately. (More likely it's less than 0.5%.) The other 95% will either halt, catch fire, and reorient their internal moral compass, or, much more commonly, get outraged that you dared to pressure them to do that. (Whether you actually applied any pressure is basically immaterial.) 

I'll list some criticisms of EA that I heard, prior to FTX, from friends/acquaintances who I respect (which doesn't mean that I think all of these critiques are good). I am paraphrasing a lot so might be misrepresenting some of them.

Some folks in EA are a bit too pushy to get new people to engage more. This was from a person who thought of doing good primarily in terms of their contracts with other people, supporting people in their local community, and increasing cooperation and coordination in their social groups. They also cared about helping people globally (donated some of their income to global health charities + were vegetarian) but felt like it wasn't the only thing they cared about. They felt like often in their interactions with EAs, the other person would try to bring up the same thought experiments they had already heard in order to get rid of their "bias towards helping people close to them in space-time". This was annoying for them. They also came from a background in law and found the emphasis on AI safety offputting because they didn't have the technical knowledge to form an opinion on it and the arguments were often presented to them by EA students who failed to convince them, and who they thought also didn't have good reason to believe in them. 

Another person mentioned that it looked weird to them that EA spent a lot of resources on helping itself. Without looking too closely at it, it looked like the ratio of resources spent on meta EA stuff to directly impactful stuff seemed suspiciously high. Their general thoughts on communities with access to billionaire money, influence, and young people wanting to find a purpose made them assume negative things about EA community as well. This made it harder for them to take some of the EA ideas seriously. I feel sympathetic to this and feel like if I wasn't already part of the effective altruism community and understood the value in a lot of the EA meta stuff, I would feel similarly suspicious perhaps.

Someone else mentioned that lots of EA people they met came across as young, not very wise, and quite arrogant for their level of experience and knowledge. This put them off. As one example, they had negative experiences with EAs who didn't have any experience with ML trying to persuade others that AI x-risk was the biggest problem.  

Then there was suspicion that EAs, because of their emphasis on utilitarianism, might be willing to do things like lie, break rules, push the big guy in front of the trolley, etc if it were for the "greater good".  This made them hard to trust. 

Some people I have briefly talked to mainly thought EA was about earning to give by working for Wall Street, and they thought it was harmful because of that.

I didn't hear the "EA is too elitist" or "EA isn't diverse enough" criticisms much (i can't think of a specific time someone brought that up as a reason they chose not to engage more with EA). 

I have talked to some non-EA friends about EA stuff after the FTX crisis (including one who himself lost a lot of money that was on the platform), mostly because they sent me memes about SBF's effective altruism. My impression was that their opinion (generally mildly positive though not personally enthusiastic) on EA did not change much as a result of FTX. This is unfortunately probably not the case for people who heard about EA for the first time because of FTX - they are more likely to assume bad things about EAs if they don't know any in real life (and I think this is to some extent, a justified response).