I think the best argument for this is developing the social skill of being able to interact with people who are different from you. When you give money to a panhandler, it's an opportunity to have an interaction you wouldn't have had otherwise.