This post is for discussing how to make really simple tests in order to spend as little resources as possible while still acquiring valuable feedback for what to focus on. It is inspired by ideas from the Lean Startup movement. There are a lot of ways to make tests that do not require a lot of money or time, for example when considering important choices regarding career paths, projects or what to study.
Note that I am not an expert in entrepreneurship. Also note that in the spirit of this post, this post is itself a small experiment. I forced myself to not take more than a few hours to write it, and if it gets enough attention or appreciation, I will know that it is worth writing more about it.
If you have advice regarding small experiments, literature recommendations, or other thoughts or feedback on my post, please comment!
Pretotyping is a very powerful technique. It is often used for testing product ideas. The basic idea is to, instead of making a product, test its potential without making it. There are many ways to do this, and I will cover a few of them here, and list examples. The person that came up with the concept, Alberto Savoia, wrote a short book about it, Pretotype It.
A few of my pretotype examples include tricking people. Temporarily. I don’t like dishonesty, and you should tell the truth afterward if you actually trick anyone.
Another useful technique is to simply rank options in several dimensions, and simply add the scores for every dimension. Do this for all options, and compare them. It sounds simple, but it is surprisingly powerful. I first read about this in Thinking, Fast and Slow by Daniel Kahneman, which is a must-read. Here are a few examples.
Kahneman suggests that a good rule of thumb is using about six dimensions.
This method can be used for any important decision including several options. The method is more accurate than simply ranking the options without specific dimensions in mind, and after you have ranked the options by your different dimensions, Kahneman explains that you will likely make a better decision even if you don’t add the scores. You will have gotten a more comprehensive and accurate view of all options. There are more good examples of using this method, and using algorithms in general, in his book.
Also remember that for some decisions it might not be a good idea to use this technique. When you reason about something, you might get a reasons-generated attitude change. It means your attitude changes to match your reasoning. This attitude change might be temporary, and if you make a decision with a temporary attitude you might regret it. For any decision where your attitude is the most important factor to you (perhaps relationships, or a new car), intuition might be a better tool for decision making.
It can be hard to get honest answers from people. If you say, “Hey, I am thinking about doing [insert something here]. Do you think it is a good idea?” you will probably get something like “Oh I love that idea!” or “You should definitely try it out!”. People can be way too supportive. You want to know why it is not a good idea, if it indeed is not a good idea. However, pretotyping or designing small experiments might take longer than simply asking the right person. The solution is The Mom Test. The Mom Test is a book by Rob Fitzpatrick, but I will refer to The Mom Test as a technique.
The Mom Test is a technique for getting honest answers. Just do these three things when you are talking with someone:
To get honest answers, do not tell the person what you want advice for. Don’t ask leading questions. Give information when needed instead of giving suggestions, stating hypotheses or stating your opinions. A good rule of thumb is to talk 20%, listen 80%. I suggest reading The Mom Test for elaboration.
If you consider a certain job, the commonly accepted test method is to work temporarily in that job. You could do it for a few months, or get an internship position, or something similar. But you can find out if you are a bad fit for the job way faster.
A good experiment should be behavior changing! If you don’t expect an experiment to change your decisions, don’t do it.
Experimentation almost always beats asking for people’s opinions, even if you use The Mom Test, but talking with people is often faster.
Remember to design experiments that give as objective results as possible. If you can interpret everything to match your current preferences, you have failed.
Remember, any result is a good result. The results telling you that your idea is terrible are the most important. The harder a result is to face, the more important it is. You want to change direction as soon as possible if the idea isn’t good, or you will not be altruistic effectively. I also recommend reading Conservation of Expected Evidence, a LessWrong post by Eliezer Yudkowsky, to understand test results better.
I find it useful to list assumptions and hypotheses regarding things that I want to test. A suggestion is to write a question for every assumption and hypothesis that you come up with. If you have an assumption that a project you want to do will take about three months, your question could be “How long does it normally take to do this type of project?” (Use the outside view!) Then rate every question by how important it is and how easy it is to design a test for it and find an answer. Then start with the questions that scored highest!
Now when you have your questions, use techniques like pretotyping or The Mom Test, and design tests for each question. Remember, REALLY SIMPLE TESTS. The best tests are the ones that take minutes. The tests should not take more than a few hours if it can be avoided. And I don’t mean the time until you get the test results, I mean the time it takes to set up the test. If you for example use The Fake Door with a website, you might have to wait for getting enough data, but the website should not take long to make.
Now you are ready to examine the world with greedy eyes, looking for the absolute simplest ways to figure out what to do!
If you found this post useful, or have examples of when you used simple tests for something, please comment!
Loved this! And congratulations on your first forum post!
One thing I like to think about with experiments is that they can help motivate you to try new things that seem daunting. Rather than something like: "2023 is going to be my year of going to the gym / going out more / learning French", you can rephrase it as, "I am going to experiment with going to the gym solidly for the next 4 weeks and then re-evaluate".
Not only does it make things more concrete and achievable, but there is also no way to "fail" - since all you are doing is testing a hypothesis
Interesting perspective! When I wrote it I thought mostly about saving time, but I also believe it is good for motivation and handling the sunk cost fallacy.