This is great! Curious what (if anything) you're doing to measure counterfactual impact. Any sort of randomized trial involving e.g. following up with clients you didn't have the time to take on and measuring their change in productive hours compared to your clients?
Unfortunately, I don't have an easy control group to do such a trial. I do my best to take on every client who I think is a great fit for me to help, so there isn't a non-coached group who is otherwise comparable. Additionally, as a for-profit business, there's an understandable limit to how much my clients are willing to humor my desire for unending data.
As of October 2020, I have had 1500+ coaching sessions with 200+ clients. Client surveys continue to show increasing benefit from my previous impact evaluation mid-2019. Clients reported an average increase of 25 productive hours per month during coaching and valued the grant equivalent of four sessions of coaching at an average of $6,857.
I’ve explored a number of ways to scale the impact of coaching over the past year and a half, including starting my blog to write about productivity. While I expect one-on-one coaching to remain my main focus, I intend to continue looking to scale in 2021. In particular, blogging and courses interest me as opportunities to reach a larger audience.
EA Coaching helps people working on the most pressing problems excel at their job.
It was founded and run by me, Lynette Bye. I’m a productivity nerd who’s been coaching effective altruists since 2017. Prior to founding EA Coaching, I studied psychology at Harvard University and researched self-control under Angela Duckworth at the University of Pennsylvania. My interests include how humans excel, self-experiments, and generally tinkering with how to live the good life. I write about these on my blog.
I work with professionals who already accomplish a lot -- consultants, professors, software engineers, CEOs, researchers -- to pinpoint their bottlenecks and help them solve the biggest problems holding them back from accomplishing more. In practice, this looks like lots of individual experiments, setting goals, and prioritizing. In 2020, I also started doing more career coaching helping people structure big decisions, gather information, and follow through on their new goals.
Most of my work is with clients who are attempting to contribute to top cause areas, since marginal improvements in productivity for this group may have a disproportionately large impact on the world. A third of my clients are at mainstream EA orgs such as the Open Philanthropy Project, Future of Humanity Institute, and Centre for Effective Altruism.
I continued to work on improving the coaching during the rest of 2019, then shifted more of my attention to writing and exploring ways to scale in 2020. Despite this, my scores on client survey metrics have largely continued to improve over the past year and a half since I wrote my previous evaluation.
I spent the first half of 2020 exploring a number of scalable options to expand the potential impact from coaching. In the second half, I decided to double down on writing as a potential form of scalable impact and a potent learning tool for myself.
Over the past year and a half, I have:
While I expect one-on-one coaching to remain my main focus, I intend to continue looking to scale in 2021. In particular, writing and courses are of interest to me.
In 2021, I’m interested in:
I wrote an impact evaluation in mid-2019, which you can read here. For efficiency’s sake, I’ve omitted repeating caveats and thoughts already summarized there.
Here are the plans for measuring impact that I wrote last January as an informal pre-registration. All of this data is collected from pre- and post-coaching surveys. I used to give the post-coaching survey after the first four calls; now I usually give it to the client when they are ceasing coaching. I also administer a pre-coaching (baseline) survey to clients about to start coaching. The data in this section is from the surveys administered between the last impact evaluation and October 2020.
I take the numbers below with a grain of salt reserved for all numbers that express more certainty than they can possibly have. When possible, I’ve put similar measurements in comparison to one another so you can triangulate for yourself how much added value you think they indicate.
I place more weight on the positive trend of scores over time, and the client survey data indicates the coaching has become more useful since my previous impact evaluation.
Summary of client survey data:
NPS=36, n=61
Decreased 8% from 2019 Impact Eval (39)
If you were more productive, how many more productive hours would you estimate you had over the past month because of the coaching?
n=57, range 0-100
Increased 52% and 60% respectively from 2019 Impact Eval (average 16.4 and median 12.5)
How many additional hours did they report?
n=24, range -40-130
Clients reported total productive hours for the past month on the intake form (pre-coaching survey) and again on the feedback form (post-coaching survey). These numbers were calculated by subtracting the pre-coaching hours from the post-coaching hours for each person who filled in both surveys, than averaging.
This calculation has a wider range than the client-reported increase in productive hours, but the averages are similar and the medians are identical.
If I instead calculate aggregate extra time by averaging the hours across everyone who filled in respective survey, I get an average of an average of 97 hours on the pre-coaching survey and 132 on the post-coaching survey, which means the difference is 35 hours. I think this method is less accurate than the above graph, however.
I only had two clients who reported tracked time on the pre- and post-coaching surveys, so I wasn’t able to control for tracked time.
By what percent would you estimate coaching changed the amount you accomplished over the past month?
n=45, range 0%-175%
If you could have gotten a grant instead of the most recent four sessions of coaching, how much money would it take for you to be indifferent between the grant and coaching?
n=59, range $80-$100,000
Increased 351% and 25% respectively from 2019 Impact Eval (average $1,522 and median $800)
How much would it have been worth to pay for the past four sessions, if you were paying out of pocket?
n=22, range $100-$3,000
I sent an annual user survey to my email list and some former clients at the beginning of January. Topics covered coaching, my blog writing, and events I held. I did a $75 raffle as a thank-you for participating. The following data is from that survey.
Summary of client survey data:
To keep the survey short, I only included the hours and Net Promoter Score questions from my usual surveys.
NPS=9, n=32
If you were more productive, how many more productive hours would you estimate you had over the past month because of the coaching? If you have ceased coaching, please report the most recent month (not when you were still doing coaching).
n=27, range 0-80
Extra productive hours after ceasing coaching
I was particularly interested in the lasting effects of coaching after the client ceases coaching. I analyzed the above data excluding anonymous responses, current clients, and clients who recently ceased coaching. This graph represents the lasting benefit to clients who ceased coaching more than a month ago and up to two and a half years ago.
n=14, range 0-36
Extra productive hours during coaching
I also analyzed the above data only including current clients and clients who ceased coaching in the past month. The result was surprisingly consistent with the data from client feedback forms at the end of coaching.
n=8, range 8-80
Which, if any, of the following types of benefit have you experienced because of coaching, either when you were doing coaching sessions or after you ceased coaching?
I added this question to assess which areas clients felt coaching helped them with most often.
Additionally, as I shared in my 2019 impact evaluation, I think that helping clients prioritize is likely the bulk of my impact, rather than adding extra productive hours. According to this data, prioritization is cited as a benefit of coaching more than twice as often as working more.
Have you personally found the blog valuable? Have you done something differently because of it?
18 of 43 readers said they had done something differently with a positive effect because of reading the blog (the others had not done anything differently).
NPS=4, n=43
If you were more productive, how many more productive hours would you estimate you had over the past month because of the blog?
n=16, range -0.1-20
Over the past year and a half, a smaller fraction of people responded to the survey, which means that biases in who responds could have a larger impact (e.g. if the top half of clients were more likely to respond, this would inflate the reviews).
Michael Townsend
EA Coaching with Lynette had significantly increased the amount of focused hours I could achieve each month, as well as providing a helpful framework I can continue to use while no longer receiving coaching.
Jason Bono
I was looking to switch from fundamental research in a somewhat esoteric field to something with more direct societal value, but wasn't gaining any traction. Lynette taught me how to better evaluate and prioritize activities. I applied those skills to evaluating career paths, application and interviewing strategies, and even in unrelated domains. After a few months, I started seeing large, accumulating dividends. I reached my original goal, and I expect to utilize what I learned to better navigate career and life decisions in the future.
Josh Curtis
Lynette helped me notice that my sleep was poor and helped me create and follow through on a plan to fix it.
Chris Bakerlee
Lynette provided me tools, wisdom, and feedback that helped me formulate better habits and make the most of my time. A significant counterfactual value-add to my life!
Amanda Hungerford
Lynette used easy-to-implement techniques to help me become more productive. The end result was significantly more efficient and focused work time. There was a noticeable uptick in the amount of work I was able to achieve, and the quality of that work. It's one of the best things I've done to improve my work.
Will
I'm a software developer looking to apply my skills to more important problems. Lynette has helped me to break down the monumental task of deciding what direction to take (and how to get there) into a series of manageable and informative actions. She questioned a lot of my assumptions about how to do this and coached me in using some productivity tools which I have found very valuable.
Bryan
Lynette introduced me to a number of tools and mechanisms to improve my ability to complete projects and do focused work. She helped me identify impediments to motivation and planning and showed me how to address those issues. I benefited from the feedback on projects and accountability of the coaching sessions.
Ben
Lynette's coaching has helped me set more realistic goals and prioritize my goals better.
Adam
Lynette's coaching helped me to nearly double my core productive hours, and to make significantly faster progress on my goals by prioritizing better.
Tyler
Prior to productivity coaching with Lynette I didn't take a solutions-oriented approach to many things that were slowing down my work. The coaching pushed me out of a naive-consequentialist frame of mind that sacrificed needed productivity optimization for the sake of direct work and created an accountability structure within which I couldn't avoid addressing productivity issues head-on.
Joshua
The only real choice that we have to make is how we choose to spend our short amount of time. Lynette brought so much clarity to the projects and tasks that I choose to take on. It's remarkable how her advice can be so pragmatic and detailed and at the same time do so much to clarify the big picture.
Oliver
There are many pressures for our valuable time. I went to Lynette with the hope that she would help me carve out my time for projects that really mattered to me. Lynette delivered that and more! She restructured my time and provided a tool-kit to stay motivated. Moreover, the coaching has provided me with a solid foundation to be highly productive in future endeavors.
The data in this section is from the pre-coaching baseline surveys administered between the last impact evaluation and October 2020.
Age, n=46
Average Age 27.4
Median Age 27
Range 20-49
Highest level of education completed, n=48
Gender, n=48
Interested in hearing more updates? Subscribe to EA Coaching’s newsletter to get more posts delivered to you.
Great write-up. Thanks for doing this, and thanks for your work!
You mentioned that "Over the past year and a half, a smaller fraction of people responded to the survey," but I don't think you report what those percentages are? The closest I can see is "200+ clients" and all the various n values, which seem to peak around 60. But you don't report your current number of clients, so I can't take much from these data. Can you report approximately what fraction of people responded in 2020 vs previous years?
I just checked, and 43% of clients who started coaching in 2020 filled in the survey, compared to 81% of clients who started coaching in 2018.
I see, thanks! Any ideas why? Maybe just because you have more clients now, and larger groups tend to have lower response rates for this sort of thing?
I'm guessing it's mostly because I put less emphasis on them filling it out. When I started coaching, I got more information from new data than I do now, so I put more effort into getting as many people as possible to fill it out. Additionally, I got feedback that it seemed strange paying clients were spending so much time giving me feedback. So now, e.g., I haven't been following up as much if people don't fill it out, and the ask is probably easier to ignore.
Larger groups, coaching busier clients on average, and only asking at the end (instead of also after the first four calls) might also contribute.