What do you need to know to make a decision, and when should you try to find out more?
I think this is a really important topic, but when you're making a decision, it's one that deserves very little attention - it's just rather important that it gets that bit of attention.
I wrote my dissertation on the topic, and would argue that basically all questions that matter should be approached using Value-of-Information (VoI). And there are complex quantitative tools that are useful for decisions about this. In doing my dissertation, I tried to actually use some of these methods. I came to the conclusion that you will all live far happier lives if you never actually use them, or even read a dissertation about them. Instead, there are some simple approaches and heuristics that get you almost the whole way there, so I'm writing this post.
Hopefully, this post can help get most of the value at a very low cost. But it will help even more if you use it as an exercise for a specific decision you are making. If possible, pick some personal decision you need to make now - even if it doesn't help, it will be useful practice.
VoI for Smart People
You have a decision to make. The fact that you are trying to make a decision implies that the answer isn't obvious - you're unsure about something. Given that, the first step is to spend 5 seconds thinking about whether getting more information would help you make the decision. At this point, you almost always should be able to think of something that you would want to know. (If the answer is that you don't need any, you probably want to think again - or read the above linked post.)
If you think of information you can get easily and at very low cost, get it. Afterwards, if the decision isn't made, come back to the post and start over - is there anything more?
If you discover there are uncertainties you'd like to understand, but it's unclear what you need to know, where to find out more, or how hard it will be to get the information, you should now spend another five minutes, by the clock, really thinking about what you might want to know. But before you start, there are a few ideas to consider.
For that five minutes of thinking, you need to think about what you are trying to decide and how information could help. Here's a set of questions to ask yourself during those five minutes:
1) What decision(s) are you making?
2) What do you not know, specifically, which is important for the decision?
3) If you knew more, would you change your decision? (If not, it's not useful for the decision.)
4) Is there a way to become less uncertain? What could you find out that might help?
5) Is the information that would change your mind worth the cost of gathering it? (This might be tricky, but see below.)
For the last question, usually the answer is obviously yes or no. Sometimes, however, it's unclear, and you need to think a bit more quantitatively about the value of the information. If you want to see the math for how VoI is used in practice, here are some examples, and some more, of how to do the basic quantitative work.
Sometimes, however, you will realize that you need to spend time thinking about the questions and consulting with others before you can start putting numbers together. That's a bit more complex, and if you want to read about how I recommend doing this in great depth, feel free to read the last chapter of the dissertation.
(1) Justifying this would require a much more in-depth discussion than this post warrants, so feel free to ignore the claim.
(2) Specifically Bayesian Markov Chain Monte Carlo simulation, and Discrete Bayesian Networks. It's all in the dissertation - which you shouldn't read (see below.)
(3) If you want to put yourself to sleep, feel free to read anything other than the first 5 pages of the fifth chapter of the dissertation. (4)
(4) Really, only pages 159-164, here. That little bit will provide 90% of the value. That's the part where I point out that doing the basics is a better use of your time than quantitative modeling. Trust me - the rest of the information in the dissertation is low-value, and for almost everyone, it's not worth the cost of reading.