While meditation works for some people, I think that the narrative of meditation will work for everyone if you just stick with it is false. I don't have rigorous data to support this; I only have my only experiences and observations. Some people take to it very easily. I gave meditation an attempt multiple times during different periods of my life (joining meditation events, short classes for an hour or two each week, yoga teacher training, and lots of shorter mediations as part of yoga classes). I never really got anything out of it. I honestly don't think that it benefited me at all.

So my non-expert, layperson's perspective would be this: just because mediation is a useful tool for some people, doesn't mean it is going to be the right tool for you and for what you want to do. If you want increased calm, self-awareness  and wellbeing, there is more than one path to get that.

Context: I have been a practicing Buddhist, primarily in the Plum Village Tradition, since 2011. In addition to my personal practice, I am currently training to become a "lay monastic" (essentially, just doing more formal Sangha-building).

First and foremost, I appreciate your openness about your experience. I definitely understand how frustrating this can be, and you are certainly not alone in this. I feel comfortable in assessing that most, if not all, struggle with this to some extent - even years into practice. Myself included. There are so many variables that impact one's ability to be present and mindful, and it is important to be compassionate toward yourself whenever this happens for you. 

Broadly, what has helped me in my own practice is not conceptualizing meditation as meditation = sitting meditation. There are many forms of meditation (walking, deep relaxation/resting, hugging, eating, etc.) and every moment is an opportunity to be mindful and present. Another aspect of this particular tradition that I have found helpful is to move away from conceptualizing meditation and mindfulness as means to an end (consequentialist/utilitarian-esque thinking), and rather, as the end in itself (this is largely how it is regarded in my main tradition, but others regard meditation as [TLDR:] the way to enlightenment). 

I have also found that more diligently studying the sutras/teachings/texts/history has deepened my understanding and helped my practice, but I have simultaneously worked to avoid over-intellectualizing; instead, working to embody the practices and to integrate them into my daily life (incrementally over time, even if it is just a few seconds of being more mindful per day/week/month). 

As for jumping into a retreat - results vary drastically. Some find this beneficial, others find it extremely challenging and occasionally harmful (the latter is often true if people are struggling with deep emotional/psychological topics). I tend to encourage people to try a one-day retreat, maybe a weekend, rather than going all in on a week or more. It may also be worth considering which type of retreat you partake in; I would recommend doing research and tuning into what sounds best based on your wants/needs at the time you are searching. I attend retreats in various traditions throughout the year, but I most often attend retreats at Plum Village centers, as I have found that style most suitable for my wants/needs. I do find that other retreats/traditions can be more helpful at different points.

It is difficult to discuss this at length in a comment format, but I would be happy to hear more about your experience and discuss it further, if you think doing so would be beneficial (others are welcome, too). Feel free to schedule some time for us to chat.

In the interim, I am happy to answer questions here, in a message, or by email.

Disclaimer: I have aphantasia and it seems that my subjective conscious experience is far from usual.^[1] I don't have deep meditation experience; I have meditated a cumulative 300+ hours since 2015. I have never meditated for more than 2 hours a day.

I've found Headspace's mindfulness stuff unhelpful, albeit pleasant. It was just not what I needed but I only figured it out after a year or so. Metta (loving-kindness) is the practice I consistently benefit from most, also for my attention and focus. It's the best "un-clencher" when I'm by myself. And it can get me into ecstatic states or cessation. Especially "open-monitoring" metta is great for me.

Related writing that resonates with me and has shaped my perspective and practice are:

1. Charlie Rogers on self-love.
2. Nick Cammarata on attachment/"clenching", meditation as a laxative and how that affects happiness, clarity and motivation.
3. A lot of the Qualia Research Institute's work is tangentially fascinating. They summarize the neuroscience on meditation quite well (afaict) and their scales of pain and pleasure map onto my experiences, too.

Maybe some of this can help you identify your own path forward?

I have a friend who did a personalized retreat with a teacher for 3 days and made major breakthroughs; i.e. overcoming multi-year depression, getting to 6th Jhana. The usual retreats are probably inefficient, it seems better to have a close relationship and tighter feedback loops with a teacher.

1. ^{^}

   I don't have an inner voice, I don't have mental imagery, my long-term memory is purely semantic (not sensory or episodic) and I have little active recall capacity. Essentially, my conscious mind is exceptionally empty as soon as I reduce external input. That doesn't mean I'm naturally great at focusing (there's so much input!). I'm just not fussed about things for longer than a few minutes because my attention is "stuck" in the present. I forget most things unless I build good scaffolds. I don't think this architecture is better or worse - there are tradeoffs everywhere. Happy to talk more about this if it seems relevant to anyone.