Illusion of predictability #2

I started a project called “the illusion of predictability” right at the beginning of my PhD. It seemed like an interesting idea. It was about questioning the possibility of using a widely featured statistical tool in economics to improve decision making and prediction; it was about merging academia with reality.

I really believed I was onto something.

I teamed up with a fellow believer, an exceptional psychologist. We used real and knowledgeable people in our experiments. They contributed voluntarily to our quest. It was both exciting and terrifying. The results were polarized and so were the comments.

I would like to thank once again to all the contributors.

My previous post featured some very insightful reactions to the project. Below are some more. Also, I list at the end of this post the main article and the comment articles that followed it, along with our reply to them.

Several recent discussions about our project in the blogosphere:

As mentioned above, now several scientists also have commented on the project through several reply papers published alongside the main article. Then we wrote our own replies to their commentaries. Here are all the discussion papers:

The main article can be found HERE

Scott Armstrong’s comments about the project can be found HERE

Keith Ord’s comments on the project can be found HERE

Nassim Taleb and Daniel Goldstein’s comments can be found HERE

Stephen Ziliak’s comments can be found HERE

Our reply to comments can be found HERE

Illusion of predictability #1

A study I recently conducted with Robin Hogarth has been featured in various blogs, including Harvard Business Review and Reuters.

They also involve some very insightful discussions.

Here they are:

Harvard Business Review – Blog by Justin Fox

Reuters – Blog by Felix Salmon

 

The invasion

Imagine that you are a general at a military compound near the border. From your vast experience, you know that when enemy troops mass at the border, the probability of an invasion towards your territory is 75%.

But…

You don’t have direct access to information about enemy troops and you have to rely on your intelligence sources. As it turns out, when an intelligence report states that the enemy troops are massing, they are really there!

Now…

Your sources tell you that enemy troops are at the border. What is the probability of an invasion?

The nice thing about this problem is that it can be applied to many fields, including business or finance, where news about an event gives clues about the occurrence of another underlying event, which is what you are actually interested in.

The answer, though, is unsettling.

No… It is not 75%.

In fact, it could be virtually anything between 0% and 100%.

When one thinks of 75%, one is making the wrong assumption that the intelligence draws a random enemy action. However, perhaps the enemy is cunning, in that the intelligence sources see their troops only when there is no invasion planned (then the answer would be 0%).

This question is taken from a 1980 article by the famous social scientist Hillel J. Einhorn and shows us how our judgments can be driven by true yet worthless and irrelevant information.

Get luckier

How lucky do you consider yourself? Very? Not so much? For most of us the answer varies from one day to another, so perhaps it does not have much of a meaning.

But from a scientific perspective, finding out about possible sources of good and bad luck would be enlightening.

It turns out, this is exactly what Prof. Richard Wiseman did in a project called The Luck Factor.

He recruited hundreds of participants, making sure that they fell under one of two categories; those who consider themselves very lucky, and those who think they are mostly unlucky. He interviewed them, got to know them in detail and subsequently formulated a hypothesis.

Then he devised an experiment to test it.

He gave everybody the same newspaper and asked each person to count the number of pictures in it. As he suspected, the unlucky group took a lot of time to do the task, while most members of the lucky group spent only a few seconds on it; they had “luckily” noticed a huge, half-page section that revealed, in writing, the actual number of pictures in the paper.

This finding links luck to attention. If you focus hard, you miss out on opportunities. Curiously, when you lose focus by doing things rather randomly, e.g. casually determine your route to work, randomize the people you meet in parties, etc… you increase the chances of receiving a positive shock that might change things for the better, without much effort.

That is not all though. So take it easy and read more about The Luck Factor, here.

The cloud

When someone tells you that a certain event “X” causes another thing “Y”, be suspicious.

There could be indeed a positive relation between the two. But, first of all, causation is a complex issue and it can go one way or the other.

It could be that actually “Y” is causing “X”.

Even if the direction of causation is well established, there is a further issue to be considered. How important is “X” as a cause of “Y”.

In their 2008 book The cult of statistical significance, Stephen Ziliak and Deidre McCloskey talk about the “oomph” effect to express the essential aspect of determining whether or not a certain phenomenon is an important determinant of an outcome given a certain context.

Consider for instance the graphs A and B. In which one would you say X is a more important determinant of Y? Although the average effect of one more X on Y is the same on both occasions, the cloud of noise is much bigger in B, suggesting that in this case most of what ultimately causes Y is relatively independent of X.

Or consider the following.

Smoking is bad for you. This is a fact. So, to a friend that, say, goes to a conference, you advise not to smoke too much.

Given the same relation between health and smoking, would you give the same advice to someone going to fight in a war?

Consultant’s luck

Say a person takes a test and scores better than 95% of people who entered the exam.

If that person takes the same type of exam again together with the same people, would he rank better or worse this time? Or maybe the same?

It turns out, there is a higher chance that he actually does worse the second time around. The reason is something called regression toward the mean.

Let me explain.

Because the result of the test depends both on ability and luck, extreme events tend to be followed by less extreme ones. Especially when chance has a lot to say about the outcomes.

So consider a firm that finds itself in unexpected trouble. The managers immediately hire a big consulting firm to save the day. And what do you know… The following month things start to look better.

The consulting firm gets paid, of course, perhaps proportionally to firm’s success. But what part of that success was due to regression toward the mean? In other words, how much of what the consultants made was due to pure chance?

The more extreme the case, the larger the role of chance.

Birth luck

How did you become the person you are? What decisions defined your current employment, your current social life, your current house, car, possessions, tastes?

Possibly, many past decisions led you to make other decisions, seize the opportunities and acquire the knowledge and wisdom that brought you where you are right now. In business and leisure, it seems we are the sum of our decisions.

Are we?

Think also about the beginning. How much would you say our birth conditions played a role in our present situation? How much did the initial conditions influence the current state of things?

One rarely asks this question.

Would I be able to enjoy the same decisions, opportunities and accomplishments were I born in Lesotho to a single mother, or in Afghanistan to a family with seven children? What if I were raised by a wealthy family, but in 1542.

How much of one’s success in business or life should be attributed to one’s decisions? I am not sure.

Given the difficulty of the question, I propose another way of looking at this issue. Consider a donation decision.  Some charity, campaign or person is asking for your contribution. Why don’t you try to base part of your decision on the initial conditions of the recipient? Was the recipient of the your contribution underprivileged from the beginning? Would she be able to have a similar life as yours, if her birth conditions were like yours?

I think one would be fairer if he acknowledges the power of birth luck.

Lottery

Say there is a lottery where the outcome is determined randomly. You are given a choice between three  tickets.

A.   24 – 28 – 33 – 41 – 45 – 47

B.   37 – 38 – 39 – 40 – 41 – 42

C.   12 – 23 – 29 – 32 – 38 – 43

Which one would you select? A, B or C? And why did you make that selection? Would you say that you chose at random?

I asked this question a few years ago to dozens of Masters and PhD students in economics, finance and management. They all knew that the numbers were randomly drawn. Yet, the option B was rarely chosen. Also, when I asked for the reasons behind their selection, only a third of them claimed to have made a random selection. Two thirds had developed a certain reason for why one of the tickets had less chances of winning than the others.

The statistically sophisticated participants of this mini experiment noticed a meaningless pattern and acted on it. Despite their wisdom, their intuition concluded that the sequence in option B was somehow less likely to happen.

In fact we tend to see patterns in many random events, which leads us to believe that some things are more predictable than they actually are. This illusion is sometimes fueled by how the information is presented to us.

Consider a different representation of the above problem.

A.     242 – 833 – 414 – 547

B.     373 – 839 – 404 – 142

C.     122 – 329 – 323 – 843

Which one would you select now? Why?

Primitive

How does the stock market work? Is it predictable? Which new firms will succeed next year? Which ones will fail? Which products will be popular? Why exactly? Will you lead a healthy life? How much control do you have over it?

More importantly… How much of all these will depend on chance? What is the weight of randomness in all of this?

Somehow, the understanding of randomness does not come naturally to us. And by us, I mean humans.

Is an outcome considered random because we do not know how it was generated? Or was it really randomly conceived? A bit of both maybe?

How does one characterize or even recognize chance events?

The problem is that, possibly, we are not yet equipped with the innate statistical sophistication that is needed to understand such issues intuitively. Humans have been living on earth for a very… very long time. The probability theory that deals with the understanding of randomness is only a few centuries old.

The first discussions about such concepts are found in the letters that went back and forth between Blaise Pascal and Pierre de Fermat. Both brilliant men, their primary concern was to decipher and possibly better perform in gambles. The letters date back to 1650s.

Hence, given the relative recency of our adventure with randomness, it might be that we will need several centuries more to evolve and fully grasp its nature and consequences.

We are still primitive in the face of randomness.

Uncertainty preference

Which is better? To know about your future: where you will be in five years, with whom, doing what. Or not knowing it, welcoming change and unpredictability in the short-term.

I am confused.

The famous Dead Poets Society (1989 film) teaches us to seize the day. Carpe diem! The message is loud and clear throughout the movie. Or is it? All the characters in the film are miserable at the end. And the poets wind up dead.

Hence organizing things so that planning will not be necessary in the future ought to be the answer. Well… I am not so sure. That sounds both boring and depressing. It might not even be possible given the level of uncertainty.

A cocktail may be the tastier option; some dose of certainty with volatility on top of it. Of course the mix would change from person to person and also in time. That is the trouble.

The question then becomes; how much uncertainty would you like in your life, right now?

Big surprise

The relation between success and creativity is complicated. Clearly creativity is considered to be a determinant of success, but the correlation is far from perfect. An idea could be impeccable, but apparently chance has a lot to say about the outcome.

How then can one predict whether or not a creative process will lead to a desirable outcome? How can one judge the quality of one’s insight and brilliance when introducing a new concept or product?

When thinking of creativity, we primarily base our judgments on how successful the idea will be. We even use creativity and success interchangeably sometimes. Is that a healthy way to look at things though?

Makridakis, Hogarth and Gaba’s book Dance with Chance is full of illuminating anecdotes on the issue. For instance it quotes instances from recent history of technology, where it emphasizes the surprise factor in creativity based success. It tells the story of Google, whose $1.6 million price tag was considered unacceptably high by internet giants, just a decade ago. Silicon valley success stories did not see coming one of the greatest ideas in their own world.

But wait a minute. Google’s insightful creators tried to sell their own super-successful idea for just $1.6 million. It seems the success of their idea was a surprise for them too.

The book provides many other examples, including how Apple’s visionary and creative leader Steve Jobs was rumored to have almost given up his stock options in 2003, which made him a multi-billionaire, just 4 years later. Somehow, he also failed to foresee his own uncontested success.

I wish you surprise yourself big time one day.

Indirect control

How much control do you have on your life?

That is a question asked by many behavioral scientists and psychologists. It is not easy to answer.

At a superficial level, the control could seem substantial. Once it is determined where and when one is born, who is one’s family etc., it feels that one has free will and lives consciously.

Or does he?

At a deeper level, one theory is pretty discomforting. What if we are mostly guided by an emotional system that we develop over the years, by experience. What if this template that we form by continuously interacting with our environment becomes who we are and dictates what we do at all times.

This would mean that if we had lived in a different environment, we would behave differently. We would become a completely different person, even perhaps a different society. Hence, it is possibly the environment that defines who we are, not ourselves.

The upside is that humans have some control over the environment. By giving it a desirable shape, we can indirectly form ourselves.

In fact, that is what we have been doing for thousands of years.

Here is an excerpt of a speech by Prof. Antonio Damasio on the power of emotions.

The whole interview can be found here.

False hopes

False hopes are everywhere. Apparently there are devices that make you thinner while you are asleep. These will only be beaten by the ones that get you skinnier as you eat.

I get emails, at least once a month, telling me to forward it to ten people if I want my wishes to come true. I remember before they used to have a context to persuade me or maybe even scare me somehow. The last one just had a tweety picture on it, yet it had passed through hundreds of wishful individuals.

Politicians use false hopes all the time and there are still fortune tellers in TV. In Spain, there are channels dedicated to them  24 hours.

I had read somewhere in the Internet that in Las Vegas, slot machines in the airports have higher stakes, designed to get peoples’ hopes up and attract them. I do not know about that, but I stumbled upon something interesting on online casinos: They are full of false dreams.

Try it out. Go to an online casino that allows you to play roulette in a virtual table with virtual money. Apply the gamblers’ fallacy: bet progressively on one color, always making sure that you cover your losses by a small margin in case you win. I tried a few years ago and in just four hours my 25 virtual dollars grew to become 4000. The virtual roulette had a memory and it wanted me to win.

I am not immune to false hopes. I ended up buying 25$ worth of chips with my real money the next day. Lost it in less than a minute.

I was hopeless.