Today’s Common Sense Investing post might be more important than anything I have ever covered. In all of my posts and videos, I tell you things that spring from one of the landmark ideas in financial economics: capital market efficiency. As fundamental as market efficiency is to good financial decision-making, it is poorly understood by most investors.

Mea culpa. I have been sharing all kinds of information based on this foundational financial principle, but I have never taken the time to explain why it is so important to your investment decisions. Condensing the foundations of modern financial theory into a single piece is ambitious, but I am going to do my best.

Financial theory starts with asset pricing. How do investors decide how much to pay for a stock? In theory, rational investors are willing to pay a price for a stock based on the present value of its expected future earnings. All else equal, investors will pay more for higher expected earnings; they will pay less for riskier future earnings.

Whether or not investors are rationally evaluating investments in this manner brings us to the concept of market efficiency. As defined by Eugene Fama in 1970, “A market in which prices always ‘fully reflect’ available information is called ‘efficient.’”

This efficient market hypothesis (EMH) sounds simple, but it is also extremely important, and terribly misunderstood.

Market Efficiency in Theory and Reality

Informational efficiency in a market stems from competition for profits, low transaction costs, and readily available information. If there is information suggesting that an asset’s value will be higher in the future, competitive traders with low-cost access to the market will buy that asset today, increasing its price. The competition to profit by incorporating new information into prices means that prices should change quickly as new information develops. New information is random, and therefore price changes should also be random.

This point is at the core of index investing, even before we consider the higher costs and risks of active management. In an efficient market, low-cost index investing makes sense, because:

If price changes are based on new information, and are therefore unpredictable, then trying to predict them – as any traditional active manager or stock-picker does – should not be expected to improve your investment outcome.

So, explaining why market efficiency matters is easy. I just did it in one sentence. Explaining how we know efficient markets reflect reality is a much bigger task. Still, it is a task worth taking. To do so, I first need to make sure we all have the same understanding of a few concepts, starting with the meaning of two key words: empirical and theoretical.

Empirical research means looking at real data, such as observing that daily stock price changes are random. Empirical observations are concrete.

A theory is an idea about why things work the way they do. Theory is used to draw insights from empirical observations. It is more abstract, but important.

In financial economics, empirical research and theory are connected by scientific method, which includes forming a hypothesis, collecting data, and testing the hypothesis. If economists make consistent empirical observations about stock returns, they can develop a hypothesis, or theory, about what is driving those returns. Once they’ve created a hypothesis, they can test it in the data. If a theory ends up doing a good job of explaining reality, it becomes a useful decision tool.

 

Random Walks and Efficient Markets

In 1900, 70 years before Fama published his landmark work on market efficiency, French statistician Louis Bachelier noticed that stock prices seemed to follow a random walk. In the 1950s and early 1960s, more empirical work emerged, supporting Bachelier’s observation that stock prices move randomly. But with no theoretical explanation for this randomness, economists concluded at the time that stock prices did not have any economic meaning.

Then, in 1965, Paul Samuelson proposed why we would expect prices to change in a well-functioning and competitive market. His “fair game model” suggested we would expect prices to change as investors’ expectations adapted to new information. Samuelson introduced meaning to the stock market’s random walk.

Samuelson’s work was followed by Fama’s landmark 1970 paper, “Efficient Capital Markets: A Review of Theory and Empirical Work”. His paper included a summary of past work, but that’s not what made it important. It was significant for its formalization of an empirical approach for testing the theory of market efficiency.

 

Imperfect Models Are Perfectly Acceptable

It’s worth noting, neither Fama nor his frequent co-author Ken French ever proposed that markets are perfectly efficient. Perfect efficiency is an ideal state that real markets can approach, but never achieve. This is not a shortcoming of the EMH, because theory is not meant to be reality. For example, even the most detailed map of a territory is still not the territory itself.

Instead, theory allows us to illustrate or predict what the world might look like in an ideal state, and then compare that prediction to reality. In efficient markets, reality does look very similar to what we would predict an efficient market should look like, which suggests our model is useful. It predicts that, in theory:

 

• Prices should move randomly.

• Active managers should not be successful at beating the market.

• Prices should change quickly based on new information.

 

Each of these predictions do describe real markets, where we have observed that:

• Stock price changes are random.

• On average, active managers do trail the market after costs. As Mark Carhart observed in “On Persistence in Mutual Fund Performance,” even managers with the best past returns are no more likely to have strong future returns.

• Event studies on how prices moved based on new information have found markets generally incorporate new information very quickly.

 

A Two-Headed Hypothesis

One of the challenges with the EMH is that it cannot be definitively proved or disproved. Fama acknowledged this challenge when he described the joint hypothesis problem. That is, any attempt to test market efficiency requires a test of two distinct hypotheses: an efficient market hypothesis and a market equilibrium model.

Stick with me. What I just said will make sense in a second.

A market equilibrium model is a model of how the market prices assets.

Think about value stocks as an example. The capital asset pricing model (CAPM) for equilibrium pricing was introduced in the 1960s. It only defined risk as the risk of the market. Under the CAPM, value stocks produced higher average returns than would be expected based on their riskiness. As such, they seemed to violate the EMH by creating a systematic excess return that could not be explained by risk.

This result meant one of two things: Either markets were inefficient, or the model being used for market equilibrium was flawed.

Which was it with respect to value stocks? The joint hypothesis problem makes any claim about market inefficiency an interesting point of discussion. If someone states that a market is inefficient, the response should be, “Under what model of market equilibrium?”

In other words, the joint hypothesis problem suggests a model for market efficiency is only empirically useful if it’s paired with a good model for market equilibrium. Put another way, we cannot test whether information is being priced in as we would expect unless we also have a good model for how a rational market prices assets.

The CAPM equilibrium model had been the grounds for much of the work in empirical finance since 1970. But in the case of value stocks, subsequent research revealed that the single-factor CAPM for market equilibrium was not accounting for the independent risk of value stocks. Adding the independent risk of value stocks to our equilibrium model has since clarified: Value stocks do not violate market efficiency; they are just riskier than we originally understood.

The 1992 Fama/French three-factor model improved on our equilibrium model for asset pricing by identifying and including the independent risks of the market, small-cap stocks, and value stocks. The Fama/French 2014 five-factor model added the independent risks of stocks from companies with robust profitability, and stocks from companies that invest aggressively.

 

Efficient Markets: From Stories to Meaning

Without the EMH, empirical finance would just be a collection of anecdotes. Using efficient markets as a framework has allowed financial economists to evaluate theories by their rejectable predictions, as opposed to observing the individual outcomes of successful investors. In other words, EMH offers us insights into how markets work, rather than having to rely on isolated anecdotes we hope to replicate.

An anecdote is like a story. It’s one sample of a successful outcome with no theoretical explanation or empirical corroboration. In a social science like economics, we want to build an understanding of the world that allows us to make better decisions, while avoiding the types of biases that anecdotes often promote.

For example, Warren Buffett’s successful stock-picking track record is an anecdote. But it’s unproductive to ask why he, in particular, was successful. It’s like asking a doctor why your grandpa lived to be 98 years old, even though he smoked a pack of cigarettes a day. Science cannot explain each and every outcome, but this does not make it a good idea to start smoking, or stock-picking.

Interestingly, Buffett’s performance has now been explained within the framework of an efficient market: He simply knew which types of risks to maintain exposure to, and used extreme discipline and leverage to do so.

Again, we see the joint hypothesis problem: When we use an appropriate model for equilibrium pricing, Buffett’s performance does not prove markets are inefficient. In fact, as our market equilibrium pricing model has advanced over time, it has gotten increasingly difficult to find violations of the EMH. With the Fama/French five-factor model, the vast majority of differences in returns between two diversified portfolios can be explained by differences in exposure to the model’s independent risks.

Raising the Bar on “Beating the Market”

You may be wondering whether adding more factors to a model for it to explain more returns may seem like overfitting or data mining. But the way this model was developed is another critical aspect of modern financial theory.

In their 2006 paper, “Profitability, investment and average returns”, Fama and French documented the body of empirical work exploring 3 common characteristics of stocks that predict higher average returns:

1.    Price relative to book value: Cheaper stocks have higher average returns.

2.    Profitability: All else equal, stocks in profitable companies have higher expected returns.

3.    Investment: All else equal, stock in companies that conservatively reinvest profits have a higher expected return.

They then took these individual empirical observations and used a valuation theory framework to construct a new set of tests.

Past work had observed these 3 characteristics individually, but valuation theory predicted there should be a relationship among them. Based on this theory, the only way to properly observe the effects of each characteristic was to control for the other two. Fama and French verified this empirically in 2006, which eventually led to their five-factor model, the most complete market equilibrium model to date.

This takes us to 2014, when Fama and French published their paper on the five-factor model. In other words, we are not talking about ancient history here.

Equilibrium pricing is a theoretical model for how the market prices assets. The five-factor model suggests the market prices more risk into the types of assets identified in the model. It can be used to empirically test the concept of market efficiency – subject, of course, to the joint hypothesis problem described above.

In this context, the five-factor model can be used to explain over 90% of the difference in returns between any two diversified portfolios. This leaves very little room for arguments favouring market inefficiency. Any return difference that appears to be an anomaly – like the returns of dividend-paying stocks – can most likely be explained by exposure to the risk factors identified in the model.

It’s important to understand that this was not an exercise in blind data mining. Our conclusions come from research based on making empirical observations, developing logical hypotheses, and testing the hypotheses. In other words, they’re based on scientific method.

Sensible Investing in Efficient Markets

Today, empirical testing for efficient market theory and equilibrium pricing is so robust, we can explain almost any difference in returns, over any historical time period, in any country for which we have data. We also can explain anomalies like low-beta stocks, and even the performance of Warren Buffett.

In short, what used to seem like “proof” that markets were inefficient has now been explained by how assets are priced in an efficient market.

Asset pricing is extremely important to how investors allocate their capital. The price you pay for a stock defines your expected return. If prices were consistently wrong, you could not know where to allocate your capital. Fortunately, there is a strong theoretical and empirical case that markets are efficient; they price stocks based on their expected future earnings and the riskiness of those earnings. Market efficiency has sweeping implications on how you should invest your money.

At a minimum, market efficiency should push you to favour low-cost index funds. There is no consistent way to exploit random stock price changes to generate higher average returns, so taking what the market has to offer is a smart bet.

At a more advanced level, there is a strong theoretical and empirical case that the five-factor model is a good market equilibrium model. As such, it could be sensible to allocate more capital to asset classes for which the model predicts higher expected returns.

Finally, I think it is worth mentioning the role of evidence-based decision-making. The medical field has a hierarchy for ranking evidence based on its unbiased quality. The lowest form of evidence in this hierarchy is expert opinion, while the highest is systematic reviews of randomized controlled trials.

In financial economics, we do not have randomized controlled trials. But we do have a nearly unlimited pool of data for empirical testing, enabling a scientific approach to efficiently accessing capital markets.

Despite this, investors are often quick to heed the “expert” opinion of a successful hedge fund manager or banker, while being quick to dismiss the kind of theory and evidence we have discussed today. Again, just as a map is not the territory, our models are not perfect reflections of reality. But if you have a choice between making decisions based on rational reason and scientific method, versus hope and opinion, I hope you’ll favour a data-driven methodology.