This is such a hard question! Even the most dedicated altruists make some carveouts, and I don't know of any consistent way that people decide what/how big those carveouts should be.
Some past writing on this:
Friendship as a sacred value
You have more than one goal, and that's fine - as you can probably tell, I've mellowed some since my mid-twenties when I was interviewed in Strangers Drowning.

I really enjoy the question you're asking here. I'd say that you make a good point, and on the surface, it might seem like there's a stronger argument for not giving a moral pass for family time, hobbies, etc.  However, I would imagine this is counter productive in the long run, ironically making it immoral. 

EA seeks to relieve suffering, but why? Just so we can float around and exist? No. In hopes that beings may have fuller lives, to enjoy loved ones, to maintain the freedom and the potential for joy that we should all have a right to. Human history has often shown that when we eliminate these fruits of life, it drives us crazy, and we create unnecessary kinds of suffering for ourselves and others all over again. If we cut out all personal enjoyment, I'd say we're destined to cause more problems, even if we're cutting it out for a noble cause. Not to say we shouldn't be frequently reevaluating ourselves and seeing what other ways we can contribute to EA outside of money, we just need to prioritize a balance. 

It's like that Robin Williams quote in Dead Poets Society, "...And medicine, law, business, engineering, these are noble pursuits and necessary to sustain life. But poetry, beauty, romance, love, these are what we stay alive for". 

From a strictly utilitarian perspective, there are good reasons to spend time with loved ones, cultivate emotional bonds, and pursue personal hobbies:

1. Your own well-being counts too. A utilitarian doesn’t just care about the welfare of strangers. They care about everyone’s welfare, including their own. If living altruistically leaves you miserable, then by utilitarian standards that’s a loss of value, not a gain.
2. Not all goods are fungible. Money can be redirected to different causes, but emotional attachments and sources of intrinsic motivation aren’t as transferable. So it doesn't make sense for a utilitarian to demand that people rewire these aspects of themselves. Human being just don’t work that way.
3. Burnout is real. A person who devotes themselves single-mindedly to a cause might be admirable, but they risk exhaustion and loss of motivation in the long run. And conversely, a person who feels joy about life will be more motivated to continue working towards the ends they care about.
4. Role models can inspire others. If utilitarianism looks unpleasant, fewer people will want to adopt it for themselves. But if people see utilitarians living rich, balanced lives, they’re more likely to be inspired by example and join in.

That said, I don't think EA is committed to utilitarianism. Instead, I think EA is more centered around beneficentrism, the idea that it's really important to help others. The difference is that beneficentrism doesn't entail maximizing the world's total welfare. Rather, it's consistent with this view to be partial to one's family and loved ones and to have carveouts for one's own personal projects.

I recommend reading Railton, P. (1984). Alienation, consequentialism, and the demands of morality. Philosophy & Public Affairs, 134-171.