I think my take would vary depending on the level of weird.
For slightly weird stuff, as discussed in Tessa's answer, it's not actually obvious this is a net cost – the gain in memorability and inferred bravery/independent-mindedness could easily outweigh any costs to credibility.
For very weird stuff, I think it probably is net bad for your expected impact / career prospects, so if you don't care overmuch I'd advise against it. But if conforming to conventional appearance norms is very difficult/alienating/demoralising to you, then you should probably do the weird thing. In that respect, I put it in the same category as having kids, or living in a non-standard location with worse network effects. To quote Julia Wise:
Some things I can do cheerfully. It turns out that giving up children was not one of them. Other people would have no problem giving up parenthood, but I suspect that everyone has something that would cause an inordinate amount of pain to sacrifice.
So test your boundaries, and see what changes you can make that will help others without costing you too dearly. But when you find something is making you bitter, stop. Effective altruism is not about driving yourself to a breakdown. We don't need people making sacrifices that leave them drained and miserable. We need people who can walk cheerfully over the world, or at least do their damnedest.
Now, all that said, what counts as "slightly weird" vs "very weird" will vary depending on the place you live and the sector you work in, and so needs to be evaluated on a case-by-case basis. Academia values conformity in appearance far less than, say, the US Government. And of course different kinds of "weird" will imply different things – kooky vs messy vs uneducated, et cetera.
It's also sadly probably true that if your appearance is unavoidably different from most of the people around you, that will impact how additional voluntary weirdness is perceived – though I'm very unsure how that last thing cashes out in practice, in terms of the impact of marginal weirdness on your reputation.
Depending on the situation, I would disagree with the 'it's their problem' attitude. I think pushing the boundary is fine and other people should be the ones to be more tolerant of the yet-to-be-normalised things, but that doesn't discount the fact that it would make it hard to get your foot in the door in the first place. I think after you've gained some foothold, then pushing the boundary has more impact.
For example, I'm a bearded guy, but in order to get a job in Japan, I shaved it off (to my complete discomfort), and after a month or two I asked my boss if I could grow it back over our summer break. Had I not shaved, maybe my job prospects would have been worse and I would not have had the opportunity to open my colleagues' minds.