The scientific method is perfectly applicable to investment decisions, says Morningstar’s Samuel Lee – but emotion has a knack of getting in the way.
The most successful societies entrust scientifically trained workers with the most specialised tasks, such as performing brain surgery, designing airplanes and setting market-wide interest rates.
Yet many individuals regularly entrust their fortunes to the investment equivalents of witch doctors and astrologers.
Or they take matters into their own hands for no good reason other than a vague belief that they can do it if they put their minds to it.
Unlike good scientists, they're not sceptical enough of themselves or others.
Brains and education are no panacea.
My father is a tenured professor of electrical engineering at one of South Korea's top research universities.
When he designs a microchip, he draws on his years of education, consults industry journals and relies heavily on the work of other engineers.
When he speculates in small-cap technology stocks, his efforts are far more casual and sometimes include asking me which ones I like – I always profess ignorance.
Despite admittedly sub-par performance over 20-something years of investing, he refuses to give up control and index his holdings.
It's as if the logical, sceptical part of his brain shuts down and a more animal, overconfident part of his brain takes over in all matters investing.
Would I, armed with nothing more than the efforts of a few spare hours each week, go into the chip design business to compete with the likes of Intel and ARM? Of course not.
There is a science to investment
Though you may not know them by their technical names, chances are you're familiar with the fruits of modern portfolio theory, the connection between risk and return, the theory of interest and the efficient market hypothesis.
Parts of financial theory are so integral to the practice of investing that most investors have forgotten they originated in academia.
Nevertheless, some domains are more amenable to scientific expertise than others.
The sweetest fruits of biomedicine originate from trained scientists; the best investment results don't always originate from finance PhDs.
In fact, some of the greatest investors are derisive of "scientific" approaches to investing. Warren Buffett warned, "Beware of geeks bearing formulas".
Why don't finance professors dominate investing?
The foremost reason is certainly emotion, which can consume even the finest minds.
Famed logician Kurt Gödel was terrified of being poisoned and ate only food prepared by his wife, Adele.
When she was hospitalised for six months, he starved himself to death.
Even very smart investors can kill their retirement plans when they're in thrall to their animal brains. Investing requires unusual discipline which, by definition, most people lack.
Moreover, this quality, in my experience, seems unrelated to brainpower.
Second, finance and economic researchers often don't have powerful enough tools to divine as much meaning from the available data as they'd like.
That doesn't stop them trying, though, and they often end up making astrologers look good by comparison.
Not helping matters is how their work can have profound economic, social and political implications, tempting researchers into the morass of politics, where their impartiality often withers and dies.
Third, most of the information in the markets is not quantifiable with the tools at our disposal.
By restricting themselves to hard numbers, scientific investors sometimes miss what qualitative investors see clear as day.
Franco Modigliani, of the Modigliani-Miller theorem, was puzzled that so many firms paid dividends.
Investors have known since the days of Ben Graham that dividends impose discipline on corporate managers, keeping them from doing too many stupid things.
Soft qualities such as culture and incentives matter, even if you can't easily assign numbers to them or model them in a closed-form solution.
There are good reasons to be sceptical of the things that come out of finance researchers' mouths but it's a big mistake to completely dismiss them.
I'd go as far as to say evidence-driven investing is the best approach for most investors because it's based on an efficient learning strategy.
Experience is a flawed teacher
Many investors pick a terrible learning strategy: personal experience. Experience is unreliable; coloured by emotions and the zeitgeist, it captures a period that's short by the standards of history.
Investors traumatised by the Great Depression learned that stocks were dangerous and should be avoided; investors who rode the bull markets of the 1980s and 1990s learned that stocks were unstoppable engines of wealth.
Both learned the hard way that personal experience is a flawed teacher.
A better strategy is to learn from history so you don't repeat the mistakes of past generations.
Scientific investing, at its best, is about engaging the data honestly, with the intention of learning something new, hopefully something discordant with previously held beliefs.
Science as it's currently practised has plenty of flaws but it's still the most reliable method of acquiring the truth that I know of.
What are the fruits of science as it pertains to investing? There's a lot of nonsense but also a great deal of sense: Factors are important and most investors should focus on investing in them.
Samuel Lee covers passive strategies as a strategist on Morningstar's manager research team.