I don't think there's a really an EA reason to pick a certain college. Just pick based on the normal considerations (e.g. where you'll most enjoy living; where you think you fit with the culture / most fun; quality of tutorials & housing & funding; general reputation; academics https://en.wikipedia.org/wiki/Norrington_Table).
I did physics and philosophy and went to Balliol and was happy about that.
The main reason was that Balliol was clearly the phys phil college at that time, with ~5 people studying it each year, out of a total of ~15 in the university. It also had David Wallace, who was a great phys phil tutor (though he's now left). I'd guess it still the biggest phys phil, but haven't checked. It's also well-known for PPE.
I think this was a decent reason to choose it - I appreciated having other phys phils to work with, and they tend to be an interesting bunch (and maybe some of the most naturally EA minded people out there!). I had lots of great tutors too.
I thought at the time that Balliol also does well on other factors: central location, OK housing (though not as nice as the wealthiest ones); culture a bit more lefty; decent academically; good reputation etc.
If I hadn't gone to Balliol I might have gone for Wadham for the social life or one of prestigious ones with beautiful grounds (e.g. Merton, New, Magdalen, St Johns). New seems to have a good combination of features.
It's true these ones are harder to get into than the others, but in Phys Phil they used to have a good 'pool' system, so if you don't get into your first choice college, they'll assign you to another one so long as you're above the bar in general.