The information about Harvard you received is very not true in the fields I am familiar with, and I suspect generally. The Harvard Math and CS departments never(!) enforce pre-requisites (which is not true for MIT CS). Compared to MIT, the "global" required courses will be fewer in number (even restricting to non-technical requirements) and easier/less demanding of your time. For instance, the single humanities division class you will be required to take can be satisfied by a logic course taught by a math professor, and the single social science course by a graduate-level course on game theory.

Are there any alignment research community/group/event nearby?

HAIST is probably the best AI safety group in the country, they have office space quite near campus and several full time organizers. 

(I am writing this before reading other responses to avoid anchoring)

My experience (current joint CS/Math concentrator) has been that Harvard is much more flexible with requirements than say, MIT (I don't know about the other colleges you named). There are only a handful of "core" courses (called "Geneds" that are typically very easy), a language requirement, and then 3 distributional classes (one must be Science, one must be Social Science, one must be Humanities), and then other than that... basically nothing besides your major's requirement. I have felt very free to take what I want, and it is no problem to skip ahead to grad-level courses if you feel ready.

Compare this to MIT, where (going off of what my friends there say) there are a bunch of cores STEM requirements (chemistry, biology, physics, etc.) in addition to even more humanities requirements than Harvard (apparently you have to take one a semester).

Overall, I have felt very academically unconstrained at Harvard, both in terms of not having to sit through boring requirements, and having great advanced classes available.

In my experience, Harvard has a considerably more active AI safety community than MIT's MAIA.

Many of my friends went to Harvard/Yale/Princeton/MIT/Stanford.* I also went to one of those schools. Occasionally, for fun, we compare our college experiences. Our conclusion is that there are differences between them that are meaningful but not deal-breaking. 

Given your academics interests, I would recommend Yale least. Other than that, any of those schools will give you fine preparation for a math/physics/CS degree. A fact you can weight weakly is that Princeton's (non-theory) CS program is relatively weak but its math program is extremely strong. The MIT graduates I know tend to be weaker at understanding foreign policy/economics/the arts, perhaps because no undergraduates study those subjects at MIT. I expect you will be able to take graduate classes and feel academically challenged at any of these schools. 

The thing that I've heard the most variance on is how happy people were in college, and how many smart, fun and insightful friends they met while there. The main reason I'd advise applying to more colleges is so that you can visit each of them in April and pick the one that you feel like you enjoy the most, because the social experience can be very variable (MIT tends to hyper-sort people into groups of very similar people, Princeton has eating clubs, Stanford used to have lots of interesting living groups but might be growing more boring, Harvard aiui randomizes living groups and they feel a little undifferentiated as a result).

*I don't know anything about Harvey Mudd, unfortunately. There are other schools that might be a better fit for you which you should obviously also apply to, but these are the ones that I would consider to be meaningful competitors to Harvard.