I think this is often a real tradeoff, but there are other ways of framing it that might help:
A) You should work in something you at least somewhat enjoy and have a good personal fit for in order to avoid burnout (I think this is 80k's position as well). Within the range of things that meet this criteria, some will be more impactful than others, and you should choose the most impactful one. EA frameworks are very useful for discerning which one this might be.
B) The aptitude-building approach (from Holden Karnofsky's 80k podcast episode): You should become great at something you like and are very good at, and then wield it in the most impactful way you can, which knowledge of EA is again useful for. (Even if it is not initially obvious how, most skills can be applied to EA in some way—for example, creative writing like HPMOR has served as a great tool for community building.)
If someone is unwilling to move away from a low-impact cause, there are still ways EA can be useful for helping them be more impactful within their cause. Similarly, if someone is set on a certain skill, EA can help them use it to do good effectively.