My problem with the 'weirdness as social-rhetorical calculus cost' theory is that I think it's more valuable to see social weirdness in a different way, which is as the measure of variance between your and another person's beliefs.
Weirdness, in many ways, is very very very good. We create better ideas when we a) have access to new ideas, and b) access to lots of different kinds of ideas. People with access to new ideas are often seen as weird, and you need to be weird yourself to search for or find a profusion of different perspectives, so people with better ideas are practically inevitably weird, despite all of the weird people with terrible ideas.
So, rhetorically, if you have a weird idea, it's helpful to find what is integral, universal, and generic about it, such that when you try to explain it to people they don't explode over how weird it is. An example: a friend was telling his partner about creating a house built around the tree, but planting the tree as a sapling in a container above, and then training its roots as they grew through the house so that they were effectively living inside of, or underneath and within, a treehouse, but the partner thought this was very very strange until my friend mentioned that people train vines along trellises all the time, and it's the same idea.
This would create an additional (and large) incentive for people to funnel their money through faux-charities, and to create faux-charities that'll circuitously direct it back into the pockets of the donors, either fraudulently or legally, like Owen_Cotton-Barratt said.
There are responses you could make to this (watchdog organizations that monitor the effectiveness of various charities and governmental authority, tighter laws regarding charitable incorporation) but so far as I can see they're all subject to the principal-agent problem.
Other than that I like the idea.
I have not publicized my support of Effective Altruism at this point due to a fear of appearing arrogant.
One could argue that this applies as well to any altruistic or charitable movement but that isn't true: with EA there is also the tacit and easily verbalized assumption that my method of charitable giving is more effective than and thus superior to other people's, and that I'm therefore not only more generous but also more edified and generally intelligent than proponents of Ineffective Altruism, of which there are legion.
An example: I was considering posting a comment on the Facebook thread dealing with this same issue. I didn't because I knew my friends would see it.
Another example: I originally liked the EA Facebook page in the dead of night, when the least number of friends would be likely to see it. A calculation on my part.
I've had conversations about EA with a few people to whom I'm very close, and responses have been mixed-to-positive so far, but I cannot see myself broadcasting my stance in regards this issue in any public forum.
From a grassroots/proselytization perspective, I seriously doubt that I am in a marginal minority when it comes to this qualm. I'm surprised to have not seen this criticism in the post above but would be happy to know that I am in a marginal minority on this issue. Since people telling other people about EA is ahem useful to the movement, I see this as an important and inhibiting issue that has, for better and worse, been hardboiled into the name and concept.