If you’ve got a fair amount of common sense – and you must, if you’re following my Common Sense Investing series – you probably already know that perfection is hard to find. In other words, in the search for solid answers to life’s greatest questions, you’ll usually find a lot more shades of gray, than pure-black or all-white.

So it is with investing. I believe that academic research and empirical evidence create the best path to sensible investing. I also believe that much of our academic and empirical understanding of financial markets starts with the Efficient Market Hypothesis (EMH), which proposes that asset prices always fully reflect available information.

If EMH is true, the implications are sweeping: Only in inefficient markets is it possible to regularly profit by identifying undervalued stocks or by successfully timing the market. In a wholly efficient market, there are no under- or overvalued securities. Prices always reflect all information and expectations at any given point in time. They only change based on new, unknown information – which, by definition, is randoWm. Since you cannot predict random events, the only way investors can expect to profit is by taking the risks already priced into the market. This can be cheaply done with index funds.

In today’s post, I’m going to explain why you’re best off investing as if markets were 100% efficient … even though they’re not.

Discovering Efficient Markets

Some of the earliest tests of market efficiency date back to the 1950s. Until then, most economists assumed trends and patterns were present – we just had to find them. In 1953, Maurice Kendall refuted this assumption, to almost everyone’s surprise. He used early computing power to examine the time series of stock returns, and concluded , "in series of prices which are observed at fairly close intervals the random changes from one term to the next are so large as to swamp any systematic effect which may be present. The data behave almost like wandering series."

In other words, there were no patterns; the stock returns were random.

Efficient Market Mechanics

Tests for market efficiency have become more robust over time. But before we present more evidence, let’s cover the economics behind why we’d expect the results Kendall discovered.

Financial markets are highly competitive and highly liquid. Independent buyers and sellers are continuously competing to execute security trades at the best possible price for their own self-interest. In these circumstances, we would expect buyers to apply their understanding and preferences to buy securities that seem worth more than they pay for them. On the other side of the trade, we expect sellers to believe the cash they receive will be worth more than the securities they’re selling. Ultimately, buyers and sellers converge and trade at a price they’ll both accept.

In liquid markets, the competition between these self-interested, opposite views, should generate prices that reflect all market participants’ expectations. In other words, the latest price becomes a mechanism for aggregating disperse bits of information about fundamental values and expectations from all market participants.

Every market participant wants to be the one to profit by bringing new information to the market. The challenge is, any market participant with new information cannot interact with the market in a vacuum. They cannot know about all of the other new information that other market participants may be contributing at the same time. Beyond that, they cannot know how other market participants are going to interpret their information.

The result is a market that reflects new information in relatively efficient prices, but that also makes it extremely challenging for any one trader to consistently profit by identifying and trading on their own new information.

Imperfectly Efficient Markets

Even Eugene Fama himself – who is credited with formalizing market efficiency – has never claimed markets are perfectly efficient. Under EMH, it’s generally assumed that market efficiency is an ideal state that real markets can approach, but never fully realize.

If we do not expect perfectly efficient markets, the big question is whether the inefficiencies can be consistently exploited … after costs. As investors, this is all that matters.

EMH is not a law or a fact. It is a hypothesis. It suggests the only way to consistently generate higher returns in a perfectly efficient market is by taking more risk. What do we do with a hypothesis? We test it, to either strengthen or refute it. The most common and robust approach for testing market efficiency is to study fund managers’ ability to beat the market without taking on additional risks (since under the EMH, the extra risk, rather than their own talent would explain any outperformance).

In financial jargon, this means we measure managers’ “alpha,” or their ability to generate excess risk-adjusted returns. If active managers can consistently achieve their ultimate goal of adding alpha, it would disprove market efficiency. But again, EMH does not suggest it’s impossible to earn higher (abnormal) returns before fees and expenses. It’s just incredibly difficult after costs.

Next, let’s take a look at some of the milestone studies supporting this claim.

Evidence-Based Efficient Markets

We begin with an important 1980 paper, “On the Impossibility of Informationally Efficient Markets . Its authors Sanford Grossman and Joseph Stiglitz explained that there has to be some incentive for analysts to uncover the new information that contributes to efficient markets.

The challenge is, this is only true before costs. In an efficient market we would expect the marginal cost of uncovering new information to equal the marginal benefit of bringing that new information to the market. In other words, after fees and costs, the end investor cannot expect to benefit. Beyond that, based on the disperse range of information and views aggregated into the price, we would not expect any one analyst to consistently generate excess returns. It’s a hit-and-miss proposition.

Next, the Capital Asset Pricing Model (CAPM) connected expected returns to market risk. In his 1968 paper, “The Performance of Mutual Funds, Michael Jensen examined 115 mutual funds from 1945 to 1964. On a risk-adjusted basis, he found that any advantage fund managers had was consumed by their fees and expenses. (In 1991, Fama summarized a number of other studies and reached similar conclusions.)

Over time, research did find some consistent violations of the EMH, with certain types of stocks earning higher returns than would be expected based on their level of risk. At first, these anomalies threatened to disprove the EMH. However, researchers eventually identified additional independent risk factors affecting stock prices. Including those risks in asset pricing models brought market efficiency back into focus.

This was an important lesson. If a test for market efficiency shows that a strategy does in fact earn consistent excess returns, it can mean two things: Either the market is inefficient, or the model being used to test market efficiency is wrong. I offered an example of the latter in this recent post. In it, I described how a newer model on the profitability factor was able to explain a previously unexplained low-volatility anomaly (where lower volatility stocks had higher returns than would be expected based on their riskiness). By disproving the older model, the seeming anomaly vanished.

In 1997, Mark Carhart published his landmark paper, “On Persistence in Mutual Fund Performance.” In 2010, Eugene Fama and Ken French, published their “Luck versus Skill in the Cross-Section of Mutual Fund Returns.” In it, they examined large data sets of actively managed mutual funds with the most up-to-date asset pricing models and incorporating multiple independent risk factors. In both cases, these academics again found exactly what we would expect in an efficient market: Few if any active managers produced enough excess returns to cover their costs.

In 2003, Burton Malkiel addressed the criticisms of the EMH to date in his paper, “The Efficient Market Hypothesis and Its Critics.” In it, he explained:

“I conclude that our stock markets are far more efficient and far less predictable than some recent academic papers would have us believe. Moreover, the evidence is overwhelming that whatever anomalous behavior of stock prices may exist, it does not create a portfolio trading opportunity that enables investors to earn extraordinary risk adjusted returns.”

That last sentence is crucial. Even if there are anomalies, they do not exist in a way that can be exploited by investors.

What About Bubbles and Buffetts?

All this said, what about when so-called bubbles form? When they do, can investors successfully exploit the clear inefficiencies?

Interestingly, in the same year that Fama won the Nobel Prize in Economic Sciences for his work on asset pricing, Robert Shiller was a co-recipient of the award, for his research on bubbles. A bubble is said to occur when asset prices rise above their fundamental values. If prices can become disconnected with fundamentals, it means the market is not efficient.

Fama’s response to the concept of bubbles and their implications on market efficiency is as logical as you would expect from Fama. In a 2010 interview with The New Yorker, he explained:

“It’s easy to say prices went down, it must have been a bubble, after the fact. I think most bubbles are twenty-twenty hindsight. Now after the fact you always find people who said before the fact that prices are too high. People are always saying that prices are too high. When they turn out to be right, we anoint them. When they turn out to be wrong, we ignore them. They are typically right and wrong about half the time.”

Fama’s point is that a bubble can only truly be a bubble if we can predict when it will end. Otherwise prices are simply reflecting the fundamental values of securities based on the information available at that time; prices will change randomly, as new information emerges.

Finally, no discussion on market efficiency is complete without addressing the elephant in the room: Warren Buffett. A 1984 article, “The Superinvestors Of Graham-and-Doddsville,” suggested the presence of Buffett and a handful of others like him seemed to prove that the market is not efficient. Buffett and his ilk were apparently able to generate excess returns by finding stocks the market had undervalued.

At the time, nobody questions that Buffett was generating alpha. But “at the time” is important, because in 1984 our asset pricing models were much less robust. In 2014, fund manager AQR Capital Management published “Betting Against Beta” in the Journal of Financial Economics. They demonstrated that Buffett’s alpha became statistically insignificant once they controlled for all the risk factors now known to us. In other words, Buffett does not prove the market is inefficient; he proves that taking the right risks can produce a good outcome.

A Fine Shade of Gray

So, to circle back to the beginning, we cannot prove that markets are purely white, i.e., perfectly efficient. Nor can we sensibly state that they are entirely black, i.e., perfectly inefficient. Real markets are closer to a light gray: They’re mostly efficient, most of the time. As such, an asset’s current price is generally an accurate reflection of its fundamental values. However, as Grossman and Stiglitz demonstrated in 1980, inefficiencies do occur. They must, for enough market players to be willing to continue participating.

Whether markets are perfectly efficient or not, the question for investors is whether market inefficiencies can occur predictably enough to exploit them, after costs and considering the risk associated with attempting to do so. In strict scientific terms, the EMH is false. But in practice, it’s true enough to be treated as true. In terms of understanding and approaching financial markets, it’s by far the strongest hypothesis we have.

Do you think the market is more efficient, or more inefficient? Weigh into the discussion by adding your comments to the video version of this post. Share this post with others to discover their perspective. And don’t forget to tune into our weekly episodes of the Rational Reminder Podcast as well.