Angelica Montagano

October 15, 2020

Decision Making

Episode 120: Annie Duke: How to Decide

Angelica Montagano

October 15, 2020

Decision Making

Annie Duke is an author, corporate speaker, and consultant in the decision-making space. Annie’s latest book, How to Decide: Simple Tools for Making Better Choices, is available on September 15, 2020 from Portfolio, a Penguin Random House imprint. Her previous book, Thinking in Bets, is a national bestseller. As a former professional poker player, Annie won more than $4 million in tournament poker before retiring from the game in 2012. Prior to becoming a professional player, Annie was awarded a National Science Foundation Fellowship to study Cognitive Psychology at the University of Pennsylvania.

Annie is the co-founder of The Alliance for Decision Education, a non-profit whose mission is to improve lives by empowering students through decision skills education. She is also a member of the National Board of After-School All-Stars and the Board of Directors of the Franklin Institute. In 2020, she joined the board of the Renew Democracy Initiative.

Good decision-making is a fundamental part of achieving our goals, so getting better at it would be in anybody’s best interest. Here to talk about making better decisions is Annie Duke, expert poker player and author of How to Decide: Simple Tools for Making Better Decisions, and Thinking in Bets. Annie starts by defining what a good decision should look like and some of the steps involved. From there, we explore the idea of how to accommodate the fact that our preferences change and we sometimes do not even know what they are in our decision-making processes. Uncertainty is a big part of what makes future choices difficult, and Annie talks about how it is caused by either luck or ignorance, the latter of which we can control, thereby reducing uncertainty as much as possible. Another big theme today how to know which decisions to spend time on and which not to. We waste a lot of time on choices that do not affect our happiness much, and on the other end of the scale, big choices often are hard because the different outcomes they present look quite similar. Annie gives us a few tools to deal with both scenarios. Toward the end, Annie dives deeper into what a good decision involves, talking about the need to step outside our beliefs by building an evidentiary record of the process which involves outside input. Tune in for a fascinating conversation that will help you get better at choosing.

Key Points From This Episode:

Introducing today’s guest, Annie Duke, and her work on decision-making. [0:00:16.3]
The definition of a good decision and the steps involved in making one. [0:02:51.3]
Examining probabilities of potential choices and the beliefs informing the examination. [0:05:00.3]
Factoring in the possibility of preferences changing while making decisions. [0:08:56.3]
Beliefs as formed by actions, and how to not see a change in course as failure. [0:13:30.3]
When our preferences are clear and when they are not. [0:15:00.3]
Dealing two kinds of uncertainty, one based on luck and the other on ignorance. [0:16:39.3]
How to know how much time to spend on making decisions, and the need to record the process. [0:20:49.3]
Using the ‘happiness test’ to judge decisions and free up time for the important ones.[0:27:46.3]
Choosing ‘quittable’ things to gather information for more binding decisions. [0:30:03.3]
Doing things in parallel so you don’t have to make one choice. [0:31:57.3]
Understanding that choices become hard when they present similar outcomes, thus that it is not sensible to deliberate too long. [0:32:32.3]
How to speed up the harder decisions; the ‘only option test’. [0:35:47.3]
Involving others while producing an evidentiary record of a decision-making process. [0:37:11.3]
How often we should aim to update our beliefs, making them less subjective. [0:37:11.3]

Read the Transcript:

How do you define a good decision?

Actually, that's what my next book is about, which is how to decide, which is trying to define, how do you figure out what a good decision is? Essentially, a good decision involves a process that works toward allowing you to get a better forecast of the future, allows you to get closer to what you would call ground truth, meaning, what is actually true of the world, in order to decide what options that you have available to you will actually be more likely to advance you towards your goals, than retreat away from it. Now, let me be clear, that means that what is a good decision for me maybe not be a good decision for somebody else, because it's all about advancing you towards your goals, what your values are, what your resources are. And this all needs to be informed by beliefs.

One of the key ways to spot a good decision process is that you can examine it, that you can explain what the process was that you went through, and that you can repeat it. In other words, if I repeat the same process over time, apply it to the same decision, I should expect to get similar results over the long run, applying that same decision. And that's where we can see there's a real breakdown in terms of good decisions. I think, first of all, people don't really know what the definition of a good decision is. And second of all, when you start to get into things like deciding by intuition or gut, you can see where that's problematic in terms of what a good decision process would be, particularly when we think about this repeatability or the ability to examine it and pull it apart and analyze it and think about how close it's getting you to the truth.

Can you talk about the steps involved in making a good decision?

Basically, the problem that we all have as decision-makers is that there's very little we can decide about where we have perfect information. And not only that, even if we do have perfect information, generally, there's going to be an influence of luck. So, even think about like a coin flip, would be a situation where you have perfect information, assuming you've examined the coin, but you still have more than one outcome that could occur and that's probabilistic. So, we can think of those two pieces. One has to do with just there's many different outcomes that can happen, another has to do with this informational piece, what are your beliefs that are informing the decision process?

So, let's start at the back end, which is, when you're considering an option, let's assume that we're deciding among options and we're looking at one option, what a good process would do is have you examine that option and think about, "What are the reasonable set of possibilities for how the future might unfold, given the option that I'm considering?" And I say reasonable because you obviously don't want to spend a whole lot of time thinking about whether an asteroid is going to hit the Earth. So, we want to be thinking about outcomes that are relevant to our decision, that would impact our ability to go forward in a way where it's reasonable to consider those. So, now we get a set of outcomes and now we think about what is our preference for those? In other words, what's the pay off for each of those outcomes?

We're going to obviously prefer things that have a positive payoff that advance us toward our goals, and not prefer things that cause us to retreat away from them. And then we need to do the next step, which is figure out how probable each of those outcomes is. And obviously, you need to do that because it's very hard otherwise to figure out whether on balance, the option is going to advance you toward or away from your goals, because, you could, for example, think about an option that you have where there's lots and lots of bad things that can happen, but they're all relatively low probability, and there's this great thing that can happen that's high probability. Or, you could actually, the reason why you need the combination of the payoffs and the probabilities, is that you could have some kind of option that you're considering where there's a lot of highly probable bad things that could happen, but none of them are too bad, and there's this one really great thing that can happen that's low probability but it's so great that the payoff makes it worthwhile, given the low probability.

That's all you want to consider of those both. And that gives us a sense for any option that we're considering, what the upside and downside potential of that option is. And then you can see how you can rinse and repeat that, because now I can actually compare it to other options in a way where I'm doing a real apples to apples comparison. The reason why I talked about the belief piece is that, while that sort of the steps for actually considering options and calculating out an expected value and thinking about what the downside risk is, and obviously, all of that's going to be contained in there, you have to realize that that process is built on top of your beliefs. So, your beliefs form the foundation of that whole process.

So, in that junk in, junk out, it doesn't matter how solid that process is, if you're not really examining the beliefs that are informing that process, the beliefs that are telling you what options do you think you have to consider, the beliefs that are telling you what you think your resources are, the beliefs that are telling you how things might unfold or how probable those things are. And actually, even things as simple as your beliefs will tell you what you think your goals are, what you believe your values are, what you think your preferences are. So, that foundational piece is actually really important in terms of how you're building that process on top of it.

How do you think about the preference part of the decision when your future preferences might actually be different than your current preferences? They evolve over time. How do you incorporate that into decisions?

Yeah. That's actually a really great question. One of the things I really like to do when I'm thinking about how to build out better decisions, is to incorporate quite a bit of time traveling into my decision process. You can do that in two ways. One is, you can, after you're thinking about an option and you're thinking, "Is this an option that I want to actually exercise?" It's really good to think about that option playing out over time. So, just figure out a reasonable time period in which the choice you would make would have an answer for it. Did it work out or did it not work out? This could be goal-driven like, I think I want to eat healthier, and it's a year from now and I failed to do so. So, that would just be generally a goal. But it also could be something like, I'm thinking about hiring a particular candidate for a job, and it's a year from now, and either they've quit or I've wanted to fire them.

So, doing that and looking back and saying, "How did those things occur? How did I achieve success? How did I achieve failure?" Is going to start to expose some of that change over time of what I want in the moment may be very different than what I want in the future. So, that gets built into that process. You can do other types of time-travel games, which have to do with, "Well, let me think now as I choose this option." There's obviously a possibility that my preferences may change in the future, so, what I want to think about in advance as I'm making the decision, is, "What are the things that could be true of the world or true of me, that would make me want to change my mind? That would make me want to get off of this route and actually exit this highway and now do something else?"

So, you always want to be thinking through this frame of quitableness in two ways. One is, I'm less worried about preferences changing and things like that, if I know that it's a very quitable option. Because then I can go with what I think I want now, and then the world's going to tell me, and I'm going to find out, and my preferences are going to evolve. And that's actually an incredibly important tool to think about, because one of the problems we have, which is why your question is so interesting, is that we actually don't know our preferences very well. We think we do, we imagine we do because we think we know ourselves. But I think we've all gone to a movie that we thought was going to be amazing, and then it turned out, we really don't like that type of movie.

Or, we used to like something, like, I used to really like horror films, and now I don't really like them anymore. So, our preferences can change. Or, I imagine a particular type of vacation that I might like, and when I go on that type of vacation, it turns out I don't like that at all. Or, I just actually had this happen to me, my daughter asked me to go backpacking this summer, which meant like 50-pound pack, hiking up very steep inclines, that part I thought that I would like, but I really thought I would hate the camping out part. I was like, "I'm more of a glamper, come on." And I'm old, and I've done a lot of things in my life. You would think that I would totally know this fact about myself that, no, I would totally hate to be camping.

But it turns out that I liked it so much that we're already planning our next backpacking trip next summer, and we're expanding it from two days to four days. So, there you go. But the thing about that that was really nice was that it was a pretty low-risk thing to do, where I could obviously turn back and so on and so forth, where I got to find something significant about my preferences. So, the first thing is, always think in advance, like, "How quitable is this?" And the more quitable it is, you can just try stuff out in order to learn your preferences. But the signposting issue is more, "When would I want to quit? What are the things that the world could reveal to me or that I could reveal about myself that would cause me to want to change course?" And actually think about that well in advance, and all of that allows you to gather information about your preferences.

So, you're doing two things, you're imagining, "If I turn out not to like this, why might that be?" Something that's going to expose some of those preferences issues. And then you're thinking about the quitting issue, either, is it quitable? So, I don't really care if I don't know my preferences, or, if it's not easily quitable, what are the things that would make me change course? I don't know if that answers your question, because it's very deep because we don't know our preferences very well. So, you have to go with what you believe in the moment, but then have a plan for the future, if something reveals itself to you.

It's almost like we have to just embrace the randomness, and that's part of the beauty of living, right? That things change, we do evolve, we have choices all the time.

Yeah. I mean, there's a lot of research that shows that our beliefs are very much informed by our actions, although we think that our actions would be formed by our beliefs and our preferences. But it turns out that it works quite a bit the other way. As an example, you can judge somebody's liking for a candidate by, if you come up and ask them, they don't seek this out. You ask them how much they like the candidate, they tell you, and then you say, "Is it okay, can I just put their sign on your lawn?" So, they didn't seek this out. They didn't go somewhere to find the sign. But after that sign's on their lawn, they liked the candidate more. That's just informed by the fact, what? I mean, think about it, you have to square that circle, which is like, "Well, I was willing to put a sign on my lawn, so therefore, I must actually really like this candidate."

We want to try to separate ourselves from our actions and realize that just because we've done something in the past, doesn't mean that we necessarily are going to like it in the future. I think that we will often stay on courses that we shouldn't be on, just because we've already been on them. And there's something about changing your mind and deciding you don't like something anymore, that I think feels to us like it's an admission that we were wrong, as opposed to allowing for the idea that preference has changed. Or, maybe I thought I would like it, but I didn't. Or, maybe I thought I wouldn't like it, but I did. Obviously, this is much more about like our own tastes and the thing that brings us enjoyment. There are categories where the preferences a really clear.

If you're investing in a stock, your preferences are going to be very clear, it's, do you make money or not? Because now your goal is just to have the stock go up in value. So, that becomes very simple. You can think about, there can be hiring situations where the preferences can be very clear. Like, if your biggest problem is recruitment costs and turnover, then assuming all things being equal, assuming equally qualified candidates, how you'd be judging your preference for one candidate or another would be stickiness, "How likely are they to stick with us and to still be here in a year?" So, there are places where your preferences will be extremely clear. But when we start getting into these things, the sort of personal taste and what brings us happiness, and what makes our life meaningful and things like that, those do evolve over time and I think we don't know them very well.

So, the idea with everything is, I'm going to make my best guess right now, because that's the information that I have and those are the beliefs that I hold and that's what I think I know about myself, but I'm going to do some work to imagine, first of all, the future where maybe it turns out that I didn't like this choice so much and think about why that might be, and then I'm also going to do some signposting to try to figure out when I should get off this ride, if it turns out I don't like that. And then you also have to really live this idea that it's subjective in the first place. So, just because you changed your mind, doesn't mean you were wrong.

What about uncertainty more generally? How do you account for that in the decision-making process?

Yeah. Basically, there's two places where you get the uncertainty, the first is in the idea that there's different ways that things could turn out. Even if you had perfect information, most things are very coin flip like, although generally, a coin flip is more constrained because there's a little more than two outcomes, because very occasionally, it will land on its side. But we will discount that. For most things, there's a variety of different ways that it could turn out. So, even if you had perfect information about the thing that you're deciding on, you're going to have lots of different ways that it can turn out, and those are going to have different probabilities.

The interesting thing about that uncertainty, which is just, there's many ways the world could unfold. And if there's an outcome that's going to occur 2% of the time, bad news for you, it's going to happen 2% of the time. And you have literally zero control over whether you'll observe the 2% on that particular time or not. So, the way that you deal with that uncertainty is to see it, right? The thing we don't want to do is pretend like there aren't lots of different ways that things could turn out. We don't want to pretend like we could perfectly predict whether it's heads or tails, which is what we end up doing a lot. I mean, if you listen to people on a hiring committee, they'll be like, "I know for sure that Taylor's the best candidate. And I guarantee you they're going to work out." What? That's crazy. You can't know that.

But that's what people say, "I know for sure, Taylor is a better choice than Morgan." And it's like, "Well, okay, I'm glad you know that for sure. I'm glad you know exactly how the world's going to unfold. Apparently, you have a crystal ball, and you should be making a lot of money telling fortunes." So, that's the first form of uncertainty. And the way we deal with that is just to see it. Because the better you are at mapping out what the possibilities are, and the better you are at understanding what those probabilities are, the better decisions you're going to make, the better you're going to be at managing risk. Certainly, we want to be very careful of the downside risk, particularly risk of ruin. And you need to see all of that stuff in order to make good decisions around that. So, that's the first way that you get uncertainty.

The second has to do with essentially ignorance. A lot of times we have partial information that puts us behind a veil of ignorance. It could be a very thick curtain, or it could be a little bit see-through, and it just depends on what we know. So, sometimes we'll know some of the outcomes that could occur, but not all, for example. Sometimes we know little of the probabilities of those things occurring, but we may know a little something. Like, we can usually have a guess, but because we lack information, we may have a very wide range around our idea about what could happen. If we're thinking about, for example, the probability of something happening, we may not be able to say with certainty that it's going to occur 40% of the time, we may say somewhere between 20 and 60% of the time, for example. And that has to do with a lack of knowledge.

There may be outcomes in the set that we don't know about and we can't identify. That can happen. There may be options that we don't know exist. And that comes from incomplete information. So, when we think about those two forms of uncertainty, luck, what you want to properly identify, we can see that luck is the piece that you don't have any control over. That's the piece that you have to just have a very clear view on. But the incomplete information piece, that's the part that you actually do have quite a bit of control over, that's the place where people should be focusing, and that's the place where we should realize that that's where all the cognitive bias is getting in the way, because it's not allowing us to actually approach information objectively or rationally, so that we can update our models of the world to be more accurate, which is then going to allow us to see the luck better, which is really what we care about. We want to see the luck as clearly as possible.

And the imperfect information piece doesn't allow us to see the luck as accurately as possible. I hope that makes sense.

How do you recommend to people to decide how much time goes into making the decision, but also how much time should be devoted to looking back on a decision so you can learn from your decisions? And then, a sub-question to that, is there merit in having some sort of decision journal to help manage that time travel?

Okay. So, I'm going to answer these in a weird order. Okay. I'm not a fan of decision journals, but I think that I'm not a fan of them because I have a semantic issue with it. I feel like a decision journal is like you're going through and making your decision, and then you journal it so that you have something to go back and look at, which feels to me very much like an extra step. A really great decision process will produce an evidentiary record. A really great decision process has no need for a journal, because the evidentiary record of what you thought, what was informing your decision, what your forecast of the future are, these are all things that you need in order to make a good decision, will automatically be recorded. So, I think it produces something that's journal-like.

So, that's why I say it may just be a semantic quibble, but I feel like people need to understand, journaling is not an extra step. It's not like you're going through when you're going through your decision process and then you go and journal it so you have something to look back on, the idea of producing an evidentiary record of actually going through and getting these things recorded is actually, you can't have a great decision process without doing that thing.

Yeah. So, I'm a huge fan of evidentiary record. That's what I would say. You have to be able to do that. We can circle back to that, if you want, in a second. Let me go back to your first question now, which has to do with time management, because that's actually an amazing question. So, obviously when I'm talking about, you look at an option and then you figure out what the world of possibilities are that are reasonable, and then you're forecasting the probabilities of those things occurring, and that's essentially giving you some kind of something that approximates, a somewhat cloudy crystal ball that allows you to see the different futures that could unfold, and then you're obviously doing a weighted average or an expected value and you're taking into account risk, and then you're comparing options and all of that stuff.

It may feel like, "Okay, ooh, that's really going to slow me down. If I'm not going to make any decision, it's really bad." But the interesting thing is that thinking about the world that way, that really what it boils down to is that all a decision is is a prediction of the future. But not a prediction of one future, it's a prediction of the futures. That's what we can think about it as, that once you understand that, and you're thinking about the world through this idea of, "What are my options? What's the impact of my options? What's the probability of those things occurring?" That actually generally, it's going to speed your decision-making up.

And why is that? Well, because when we think about really understanding what a robust process would look like in terms of decision-making, that's what allows us to figure out, well, when can we go skinnier? When do we have wiggle room, as we're thinking about what the future might hold, to allow the crystal ball to be even more cloudy than maybe we would like? To not quite shine it up as much. And that has to do with the ability to foresee the future. So, let's think about that. The first thing we can think about is impact. We talked about, when you're considering different outcomes, how much is that going to advance you towards your goal or away from your goal?

So, generally, what we can assume is that, if you're thinking about a particular type of decision and you realize that none of the reasonable outcomes in there are particularly going to advance you much toward or away from your goal at all, you can assume that that type of decision is pretty low impact, at which point, you've got a big fudge factor. So, essentially, you can think about it as like, "If the worst of the reasonable things that I can think about happen, is that really going to have much of an impact as I think about trying to achieve my goals?" And if the answer is no, you really don't have to spend very much time on the decision, and you can go pretty fast with those. Let me give you an example of a decisions like that.

When you look at how much time people spend on what to watch on Netflix, what to order off a menu, what to wear, it turns out, the average person, it adds up to about six to seven workweeks a year.

So, what this tells us, if we think about happiness as essentially a proxy for, are you achieving your goals? Is that, and this is what I call the happiness test, which is, what it tells you is that what you order on a menu is actually incredibly low impact, what you watch on Netflix is going to be incredibly low impact. So, there's all sorts of decisions that you can make that are going to be extremely low impact, but yet, we tend to really linger over those decisions and we spend a lot of time on them. So, the first thing is, what this tells you, you should really be speeding those up. Just don't worry about it too much, "Chicken, fish, I don't care," just pick. Because, even if it's bad, it doesn't matter. But why are you taking so long in those decisions?

And I think the reason is that they're going to cycle really quickly and you're going to get an outcome really fast. So, I'm going to get my chicken, I'm going to know what's bad. And because we don't take uncertainty into account very well, we think that if it's bad, we made a mistake. But of course, you didn't make a mistake because you don't have a time travel machine and you didn't know whether the chicken was going to be better or the fish was better. You were making a bet that you would like the chicken better. That as you thought about the chicken versus the fish, that the chicken was more likely to be something tasty for you, than the fish was. But more likely is that, doing a lot of work in that sense.

So then when you get the chicken and you find out it's bad, you think, "Oh, I should have taken more time, then I would have been more certain of my choice. I could have had a better outcome. I made a mistake." So, instead of thinking about that way, say, "It doesn't matter that I made a mistake, because this is not going to make me happy or sad. Literally, it doesn't really matter to me." So, that's kind of the happiness test. You can go even faster when something is low impact and it repeats. So, choosing things to eat or like that choosing things on Netflix, that helps to essentially mitigate the downside, if I don't like my meal, I get to order something else in four hours, or I could go to a different restaurant in a few hours, or I could come back to the same one and try the fish. If I don't like something on Netflix, obviously, I've watched a lot of stuff on TV.

If I don't like a book, I can read another book in about two seconds. So, that's another way that we can think about how can we speed our decisions that has to do with the impact? And then the other thing we can think about, which we talked about before is, we can think about the quitting part of it. How easy is it for me to get away from the option that I've chosen and go circle back to an option that I either projected or new option that has appeared on the horizon? So, we can think about, for example, the difference between dating and marrying. Marrying is a much bigger commitment, it's much harder to unwind, it's harder to quit. Dating, you can have someone call you in the middle of the date and you can pretend to have an emergency, and you can leave.

So, if we recognize that the thing that we're choosing is very quitable, then we can also fudge it. We have a big fudge factor, because, just like in the food case, if it doesn't work out, we don't really care. So, I wouldn't spend too much time trying to gather information because it's like, doesn't really matter. If I can quit, it also doesn't matter so much if it didn't work out well. If I don't like piano lessons, I can go do guitar. So, we can think about that as well, and then we can also be proactive in that and start choosing things that are more quitable, in order to specifically gather information so that when we do have to do something like marry or buy a house or roll out a marketing strategy that's nationwide as opposed to local, these are all things that are heavier lists, there's bigger costs to them, they're higher impact, they're going to be harder to unwind, that we actually think about what are the things that are very quitable that we can do in front, where we can just act really quickly in order to be big information gathering machines?

Because the faster that we can cycle those decisions, the faster that we can start getting feedback, the more information we have, which then goes back to this idea of, what are the beliefs that you have about the world? And your own preferences, and that increases the quality of that knowledge. So, that's another thing we think about as quitableness. In that optionality, which quitting is just preserving options, we can also think about exercising options in parallel, if you can do more than one thing at once, you don't have to worry as much about each thing that you're doing. So, if I get to buy 40 stocks, I'm a lot better off than if I get to buy one.

If we're in the restaurant together and you can't decide between the chicken and the fish, you could ask Cameron, if he wants to have the chicken and you'll have the fish and you'll share. So, if you can do things in parallel, again, you don't have to worry too much about it, you can exert... So, that'll be like A/B testing in marketing, would allow you to do stuff in parallel. That's another way to go faster. And then the last thing I would say is that there's a time that you can go faster that I think is incredibly counterintuitive. And that is when you have two options that are super, super close to each other, so that you're really having trouble deciding between the two. This is when they are high impact.

So, in the before times, when we went on vacations, imagine you were trying to decide between a vacation to Paris or a vacation to Rome. These are the kinds of decisions where people really get caught in analysis paralysis. And this is true on whether it's sales strategy, or two houses that you're trying to choose between, or two vacations that you're trying to choose between, or two jobs that you're trying to choose between, whatever it is, this is the place where people really get hung up. And the reason that they get hung up is that, if you think back to that idea of, what's a really good decision process? And how do you think about how you compare options? When you go through that process for Paris or Rome, what it means is that as you think about what the reasonable set of outcomes are, and what the probability of those outcomes occurring are, it looks pretty identical for both of them. That's what's happening to you.

That would be true as you are weighing two jobs, right? That kind of what they expected value is, how likely it is to actually advance you to whatever your career goals are, is looking really similar between the two choices, same thing with two houses. So, when we go back to that idea of we're always deciding what we're almost always deciding behind some sort of veil of ignorance, whether it's, again, like a thick curtain, or something that's a little bit see-through, where you've got something, but not much. The problem for us in that situation is that we have the illusion that we could parse those differences out. That somehow when we have two options that are nearly identical, that if we just spent more time, if we just gathered up some more information, and this is particularly problematic when we live in such an information-rich world, and there's all these like TripAdvisor and whatever sites, that we can somehow parse those very small differences apart.

But we're losing sight of a couple of things there. One is that the time you're taking to try to parse out these little tiny differences, is not time well spent, could be spending that time going and examining other things that are actually going to really make you happy, like spending your time on things that are actually going to advance you towards your goals much more. So, I would rather not try to pick up like 0.1% in expected value over here, when I could be over here doing something that might get me like a 2% gain. So, it's kind of time wasted, even if you could separate out those differences. But the fact is, you can't, because the information that you really need to know is how it turns out. And we cannot know that because we don't have time machines. So, this, I think, is one of the keys to unlocking how to start deciding faster, is to realize that when a decision is hard, it means it's actually easy.

In the sense that, when you have two options that are amazing and very close, you can generally flip a coin. So, how do you figure out if two options are great, great enough that you should flip a coin? You can apply something called the only option test, to just, if Paris were my only option for my fabulous vacation, would I be super-duper happy? And if the answer is yes, I would be super-duper happy, then great. If Rome were my only choice for an amazing vacation, would I be super-duper happy? And the answer is yes. Great. So, they both satisfy the only option test. If I had to spend my vacation in a meatpacking plant during coronavirus, would I be happy? No. Okay, great, well, then that's not under consideration, right? So, once we get to two options that already satisfy the only option test, what that means is that they're close enough in expected value that you probably are wasting your time trying to parse the differences out and you should flip a coin.

So, generally what that tells us is that we should be approaching decisions in terms of speed, using something called the menu strategy, which is, the real heavy lift is in the sorting. What are the entrees that I like? What are the things I don't like? What are the vacations I would like? What are the vacations I wouldn't like? What are the careers I would like? What are the ones I wouldn't like? So on, so forth. And then once you sorted it into the bucket of here are things that pass the only option test, and pretty much flip a coin among those. You're not going to be gaining very much by spending extra time there.

Can you talk more about the idea of an evidentiary record and how that can be applied to investment decisions specifically?

The part of the process that we have control over is the information that we have that's informing the decision. And we know, and this is, obviously, Conoman has talked a lot about this, Michael Mauboussin talks a ton about this, the problem with the inside versus the outside view. So, the inside view is the world from your own perspective, informed by your own experiences. Obviously, you're thinking about the world, informed by the mental models that you happen to apply to the world. And the outside view is the world independent of your own perspective. In other words, what's true of the world, independent of you. And then also the way that other people would view the decision that you're considering. The situation that you're in, how they would view it.

The reason why that's really important is that two different people can be looking at the exact same dataset, and have very different conclusions about what the right decisions would be based on that data, because data are not truth, human beings must model them. So, when we think about the inside view, that's where all the cognitive bias is going to live. Essentially, when we think about something like confirmation bias, obviously that's an inside view problem because you're trying to confirm the beliefs that you personally have. Confirmation bias isn't trying to confirm other people's beliefs, it's trying to confirm your beliefs. A motivated reasoning is reasoning about information in a way that will support the beliefs that you have, as opposed to thinking about information, independent of your beliefs, in order to make your beliefs more accurate. Even something like availability bias, like things that have happened recently to you, that are easier for you to recall as a human being, are things that you then judges as more frequent.

You can see that like cognitive bias, the thing that we're really trying to discipline in our decision process is living in the inside view. What we really want to do is get to the outside view. In other words, to start thinking about, what are the things that we know and what are the things that we could find out that we could apply to this decision? So, the first thing is that, obviously, you're naturally creating an evidentiary record, when you're saying, "Here are the different ways that I think the world could turn out." So, you're writing those down. And then you're trying to figure out, trying to make a forecast of some sort of value.

A probability of something turning out, you can set up bins, you could make estimates, for example, of what you think, if you're making widgets, how many widgets you think that... something that will be produced, for example. So, you're trying to figure out, "For me to actually map out what I think the possible outcomes of these are, and what the probabilities of those things are, I have to start examining my knowledge, but then, I also have to start thinking about what are the things that I could go find out?" So, you're looking for relevant base rates, that's obviously going to be an outside view thing, but then you should also be looking for other people's perspectives. So, how do we go about actually doing that? The first thing is, figure out what it is that you're deciding about. And this is no small thing.

Let me give you an investment choice. Let's say that you're deciding that you want to invest in a stock. And given that you're making a decision that you want to invest in a stock, I'm assuming that you think you know something more than the market does. Because otherwise, I think you would just be doing some sort of indexing, right? So, you're assuming you know something more than the market is. So, the first thing is to actually make what's implicit in your decision to purchase that stock explicit. In other words, you have to write down, "Here are the things that I think I know, that the market doesn't know." You need to write those things down, because then, that's going to allow you to start thinking about, "As I think about what the possible outcomes are and what those bins are, that is something that I need to know. What is implicit in my decision? What does it really mean, that I think that this is a stock that I'm supposed to buy?"

So, now you figured out what that is, and you've written that down, and you have made your opinions on those things explicit. So, you can't just say, for example, "I think that the company is going to produce more widgets than the market thinks." You have to say, "What percentage of the time is the company going to produce less widgets than the market thinks, the exact same number of widgets?" You have to specify, "How many widgets do I think that the market thinks it's going to produce? How often is it going to produce 5% more, 10% more, 20% more?" Figure out what it is that's important for you deciding on this thing. So, now you notice you've got a lot of evidence now. You've mapped out, "Here are what I think the different bins are, this is what I think the forecasting, the bins are. Here's my thesis," because you need your thesis.

You've done some rating of what the market opportunity is, you've done some forecasting, so on and so forth. So, it's going to be different for whatever you're investing in. And now here's the key thing, now you need to ask somebody else, hopefully more than one person. But, when you do that, you cannot tell them what your answers are, and you should not do it in a group. For example, let's say that we're on an investment committee and it really matters how many widgets we think this company is going to produce. You would define the bins, and you would independently ask the people that are working on the investment with you, whether it's a committee or you're in a partnership or whatever, to then forecast those bins. Now, they forecast those bins, I'm just giving a very sliver of an example here, and then each person will write down a rationale, no more than three sentences, "Why do I think that this distribution is the most accurate distribution."

So, now, each person has now produced this record, "This is what my belief is." Because we've defined what it is, and we've been very precise in our definition of what we think it is that we're supposed to be forecasting here. Now, it's not always just going to be like, how many widgets? You assume you're going to have the ratings on a scale of zero to five, "How big do we think the market is?" You could do it that way, a variety of things. For any investment, you can think about what those things are going to be. So, now you've got some set of opinions that you've extracted from each person and you've done it independently. And each person has written down a rationale for why they believe what they do. Okay, this is a lot of evidentiary record that we've produced now. So, now, what happens is, when you come to talk about it, everybody sees all the responses in advance of actually having a discussion about it.

Why is that really good? Well, because now, instead of talking about what you agree about, you're going to talk about what you have dispersion about. And that's what you really care about. Because, most times when we're in committee, it's all about just convincing other people that what you believe is true, and then other people want to convince other people that they also believe that thing. And most of the meeting is spent on, "I think the Earth is round, and here's why." And then Ben says, "I actually believe that the Earth is round also, and here's my evidence for that." And then Cameron says, "I also believe the Earth is round too, and here's why." And we forgot to talk about that there's a flat-earther sitting over in the corner.

Now we've got everything together, everybody's seen these very precise forecasts, we can now tell that we disagree, right? Because, I thought that there're 5% more widgets than the market is predicting, Ben was 60%, and you thought it was 35%. So, now, we each get to just convey what our rationale for that forecast is. Now, I used the word convey intentionally because I don't want to try to convince you of my side. There's no reason that we need to agree on this, which is another thing, we don't have to agree to decide, we get to decide without agreeing. This is incredibly important. And we're just trying to inform the decision. So, you can say why you believe what you do, I get to say why I believe what I do, and then we can update our forecast if we want to, and then that's the end of the process.

And all of that is now on record. We now know exactly what we believed at the time, we know that we actually disagreed about what those bins were. Of course, we do. It's subjective. We're not talking about like two plus two equals four, where we get to decide whether we're right or not, this is about trying to surface as much information as possible, and what I'm going to find out when you convey your position is that the model that you're applying may be different than mine, or you may have different data than I do that you're thinking about, and now I get to learn that from you. And what it allows me to do is get to the outside view. Because the whole problem is that we don't access the knowledge that other people have enough, we don't access the opinions and the perspectives that other people have enough. And that must be part of the evidentiary record, is getting as many of those perspectives as possible, writing them down, and then looking at the range of what people believe.

And then just like you can make a decision without knowing exactly how the world is going to turn out because it's probabilistic around that range, you can look at the different opinions that live in the group that you're talking to, and you can say, "Okay, given that, do we want to move forward with it?" Without everybody agreeing on modeling the data exactly the same, that wouldn't be true of how the world actually works for one thing or what people's states of mind are. But notice that when you do this, you've naturally produced this evidentiary record. So, there was no journaling involved. All I was doing was, I was eliciting feedback from you and I have written down my own feedback, and I had just withheld it from you.

Now, what's really beautiful about that is that, because I know that the feedback that I'm giving is going to be exposed to other people, and because I know that the feedback that I'm giving is going to be looked back at, so, there's two forms of accountability here, and when it gets exposed to other people that I am going to have to be able to give my rationale for it, I can't just say, "Because I think so," I can't just say, "My gut tells me," I have to say, "Here's why I think that the idea that it would produce 5% more widgets than the market actually predicts is 65%." I have to be able to explain that to you, not that I convince you, but that you understand why I believe what I do.

That means that when we go back to this inside and outside view problem, and how much are we able to get out of the inside view and shift over to the outside view, you're going to be much more likely to be trying to imagine the different ways to look at the problem, to go look for data that would help me to actually make a better forecast or offer a better opinion in the group, so that my rationale is better supported, so that my opinion, rather, is better supported. So that when I know we're going to look back, that I'm hopefully going to be moving toward a place of accuracy and calibrating to the market that I'm actually investing in, and that's going to help me to debias, because it's going to push me to be more accurate, as opposed to just being a good narrative spinner. I hope that makes sense.

How often should we aim to update our beliefs?

Oh, right. Always. Because I think that you need to realize, updating isn't doing a 180. I mean, sometimes it is, but mostly, it's not. So, what we want to be doing is trying to get as many perspectives on information as possible, and then always be trying to think about, "What are the gems that I can pull out and incorporate?" Sometimes you're going to do a 180, but mostly, you're going to do like a one degree. But the thing is that one degree really, really matters. Think about 1%, go, "Holy cow, that would be amazing." I mean, what if you could just pick up like 25 bips? Okay. I mean, it's a little tiny thing. So, this is the way, I think, that's really good to think about our beliefs, and then on the strongly held thing, why that gets in the way. Okay.

Our beliefs are neither true nor false, they're sitting in between. And what we're always trying to do is move them closer to, what if we were omniscient, then what is the thing that we would believe? And I think that we all know... Well, actually, I remember there was this old Saturday Live skit where someone dies and goes to heaven, and they're asking the angel, "What's the grossest thing I ever ate accidentally?" And the angel is like, "I don't want to tell you that." And they're like, "Okay, what's the 10th grossest thing I ever ate?" And it was like, "Oh, you didn't realize it, but there was a cockroach in your yogurt." That's kind of how our beliefs are. If we're omniscient, we would find all sorts of cockroaches in them. Okay.

But there's things that we don't want to reveal to ourselves, because if we knew that there were cockroaches in our beliefs, then we would feel somehow like our identity had been attacked. Because what is our identity, but the things that we believe? I don't know what else it is. This is what I believe to be true of the world and myself, and so on and so forth. So, when we feel like our belief is getting attacked, I think that it would feel like our identity is getting attacked. So, we want to step back and say, "Look, our beliefs are neither true nor false, because we're not omniscient." So, we're trying to just scoot ourselves along the way to be more accurate, which means that we have to always believe that the things we believe are up for revision. It's kind of at all times.

Now, that doesn't mean that you can't decide. Because in the moment that you decide, you have the knowledge that you have, and assuming that you've done the work that you need to do, this is the best that you can do in this moment, this is your best forecast, and you believe, you can believe completely that somebody else is in a situation where they're looking at the same data, and you're both obviously not exactly correct, but that you're a little bit more correct than the other person. And that's enough we know to win all the money in the world, win all the happiness in the world. That's enough to get you there. So, I would really recommend that, there's a book from Phil Tetlock, who people, I'm sure, are familiar with, he wrote Superforecasting, but he wrote a book before that called Expert Political Judgment, which I think is an amazing book for anybody to read.

And what it really talks about is, when you're a subject matter expert, that you get pulled down into a trench, which is your subject matter expertise, because you have such strong models of the world and such deep belief in the models of the world that you have, that when new information comes in, you actually will pull the information down into the trench with you. In other words, you view the world through the idea of like, it's in 1970s, and you view the world through a Cold War model between Russia and America, but then the world changes. In the late '80s, all of a sudden, the world changes, so then do you start to change your models of the world, or do you just start to incorporate the information into the model that you have?

And I think that you know this with investing, people end up with a particular type of mindset, whether it's like growth, or value, or trend following, or whatever, and they get very entrenched in those models. And what we want to do is actually be, Phil Tet would say more of a fox, where we can see things from all sorts of different angles and all sorts of different models in order to come up with great creative solutions. But what that means is that you need to not dig yourself such a big trench. So, the idea is that at the moment that you make the decision, you should actually have a lot of conviction in what you believe. But the conviction is that you believe that relative to the other people that you're making decisions against, that the conclusion that you're coming to is better than theirs. That doesn't mean that in an absolute sense that you believe your beliefs are the best. I hope that that makes sense.