It is a problem and there are already solutions for that in the field (Public Economics/Environment Economics). This is called the Distributional aspects of CBA, if you want you can take a look at the course Cost-Benefit Analysis from the University of Helsinki which provides related material. The link to the course: https://studies.helsinki.fi/courses/cur/otm-e45f15a4-2322-4e44-93ba-e3bbb2e95d63
The 2 research papers that they used in the course, you can take a look here:
Hope that helps!