What do you need to know to make a decision, and when should you try to find out more?
I think this is a really important topic, but when you're making a decision, it's one that deserves very little attention - it's just rather important that it gets that bit of attention.
I wrote my dissertation on the topic, and would argue that basically all questions that matter should be approached using Value-of-Information (VoI). And there are complex quantitative tools that are useful for decisions about this. In doing my dissertation, I tried to actually use some of these methods. I came to the conclusion that you will all live far happier lives if you never actually use them, or even read a dissertation about them. Instead, there are some simple approaches and heuristics that get you almost the whole way there, so I'm writing this post.
Hopefully, this post can help get most of the value at a very low cost. But it will help even more if you use it as an exercise for a specific decision you are making. If possible, pick some personal decision you need to make now - even if it doesn't help, it will be useful practice.
Ready?
VoI for Smart People
You have a decision to make. The fact that you are trying to make a decision implies that the answer isn't obvious - you're unsure about something. Given that, the first step is to spend 5 seconds thinking about whether getting more information would help you make the decision. At this point, you almost always should be able to think of something that you would want to know. (If the answer is that you don't need any, you probably want to think again - or read the above linked post.)
If you think of information you can get easily and at very low cost, get it. Afterwards, if the decision isn't made, come back to the post and start over - is there anything more?
If you discover there are uncertainties you'd like to understand, but it's unclear what you need to know, where to find out more, or how hard it will be to get the information, you should now spend another five minutes, by the clock, really thinking about what you might want to know. But before you start, there are a few ideas to consider.
For that five minutes of thinking, you need to think about what you are trying to decide and how information could help. Here's a set of questions to ask yourself during those five minutes:
1) What decision(s) are you making?
2) What do you not know, specifically, which is important for the decision?
3) If you knew more, would you change your decision? (If not, it's not useful for the decision.)
4) Is there a way to become less uncertain? What could you find out that might help?
5) Is the information that would change your mind worth the cost of gathering it? (This might be tricky, but see below.)
For the last question, usually the answer is obviously yes or no. Sometimes, however, it's unclear, and you need to think a bit more quantitatively about the value of the information. If you want to see the math for how VoI is used in practice, here are some examples, and some more, of how to do the basic quantitative work.
Sometimes, however, you will realize that you need to spend time thinking about the questions and consulting with others before you can start putting numbers together. That's a bit more complex, and if you want to read about how I recommend doing this in great depth, feel free to read the last chapter of the dissertation.
(1) Justifying this would require a much more in-depth discussion than this post warrants, so feel free to ignore the claim.
(2) Specifically Bayesian Markov Chain Monte Carlo simulation, and Discrete Bayesian Networks. It's all in the dissertation - which you shouldn't read (see below.)
(3) If you want to put yourself to sleep, feel free to read anything other than the first 5 pages of the fifth chapter of the dissertation. (4)
(4) Really, only pages 159-164, here. That little bit will provide 90% of the value. That's the part where I point out that doing the basics is a better use of your time than quantitative modeling. Trust me - the rest of the information in the dissertation is low-value, and for almost everyone, it's not worth the cost of reading.
I read (or sort of skimmed) your dissertation (since I have a decision to make---actually (too) many) and thought this might provide some useful advice on how to do it.
I was surprised you discouraged people from reading it ---maybe you are trying to hide the fun. I found it be one of the most exciting and racy PhD thesis' i've ever read ---it could be a best seller.
And it just doesn't read like a pulp novel; like the best novels, it also hits peaks of scientific inspiration which i think rival the visions I've been told you get by reading great poets or theological works or einstein or Dirac or hawking in physics --takes you to a transcendent place.
In any event while i enjoyed the read, i didn't get my problem solved---i.e. advice on what decision to make. (I 'might even have to decide first which of my problems i should try to solve first, and then do a 2nd iteration of the method and try to solve that particular problem. Or, the method might suggest I try to solve all the problems at once---eg if your house is burning down in a fire you both put it out and rebuild it at the same time.)
Since i found the procedure not exactly intuitive, it seemed to suggest you have to weigh off the cost of a 'search procedure' to get information about the possible value of alternative decisions, combined with the cost of making a decision. (eg for a burning house, you might spend some costly time consulting with some firefighters and homebuilders about the best course of action.)
A simple example might be shopping for an item. You could just go to a place you know--and you know the cost, and also the quality or value of the item. Or you could spend alot of time and money shopping around, to see if you can find something with a better value/cost ratio. There is always the risk that you may just waste alot of time and money and find nothing better.
I was wondering if the example above is the sort of decision making problem you are talking about.
(I think my main problem is finding a job---but I don't really know where to look, and if there are any of the kind I want (I might be looking for something that does not exist) , and whether and how to research finding one. The cheapest default option might be just to wait and see if one comes to me, though it may lead to same result----nothing. One problem is I don't know what the 'prior' is--any more than did the people from europe who explored the USA knew what they would find. They could try to get information from the indigneous people, but that takes time and effort,, and they didn't know how reliable it would be.)
OK, in retrospect, it definitely turns out that designing good biosurveillance systems was more important than I implied 2 years ago.