The Science Of Forecasting Long-term Oil Prices: What Not To Do

In 1992, an M.I.T. working paper that I wrote noted that forecasts of the oil prices two and three decades into the future were usually strongly in error after a couple of years. No few readers told me that they interpreted the work to mean that it was not possible to predict long-term oil prices, but that was a misinterpretation. Forecasts has gone astray because of three primary errors: a belief in the so-called Hotelling principle, mistaking transient, political problems for physical factors and a tendency towards Malthusian pessimism. (More than Malthus himself, in fact.)

The Hotelling priniciple, as some described it, proved mathematically that mineral prices had to rise exponentially over the long-term, and oil prices should rise at the rate of interest. Despite the fact that this was contradicted by history, many academics continue to defend the idea, ignoring its theoretical shortcomings as well.

Judgements that higher prices in the 1970s were due to resource scarcity rather than the oil supply shocks in 1973 and 1979 appear to be little more than reflective of a tendency for investors to think “this boom will have no bust” an attitude that goes back to the xxth century Tulip Bubble and continued recently with the U.S. real estate boom. Some argued that the oil industry merely took advantage of the Iranian Revolution to employ its monopoly power to raise prices, but too many acted as if the loss of 6 mb/d of supply, plus fears of widespread upheaval in the Muslim world as the Ayatollah Khomeini called for a struggle against other governments in the region. (Echoes of that persist to this day.)

Finally, there is the fact that nonrenewable resources are different; you can’t produce oil where is none. No one questions that automobile production will be constrained by lack of discovery of new autos; you can build a plant wherever you want. But geologists downplay the economists’ view that higher oil prices lead to more supply, as ignoring the need for discovery. In fact, economists are just aggregating the impact of discovery, just as they argue higher prices will reduce demand, without pointing to the specific investments that cause such to happen.

The peak oil crowd bristles as suggestions that they believe we are “running out of oil,” and few make that argument, but President Carter did say, “The oil and natural gas we rely on for 75 percent of our energy are running out.”   Not only did the expectations of his government prove dramatically incorrect, but the perceived need to emphasize coal to preserve natural gas remains one of the major flawed policies afflicted on the country.

Which highlights the role of analysis, good and bad. It is obvious now that our natural gas resources are abundant, and optimists like myself can be accused of relying on hindsight in our criticism, but the truth is that price controls on natural gas were clearly responsible for the shortages that occurred in the 1970s. That’s what price controls do, to paraphrase the current commercial campaign, and most economists had no doubt of this, but a desire to please consumers with cheap energy and a fear of appearing to appease “big oil” meant that it took years for the policy to be reversed.

Another error that became apparent at that time, and continues to bedevil the industry, is the influence of short-term markets on long-term forecasts. When prices dropped sharply in 1986, most predicted that they would continue to rise, but none predicted a ‘recovery’ to pre-1986 prices, even though nearly all had argued that 1985 price levels were unsustainably low. In 1998, when the price collapsed from the low level of $30 to $20 (in 2014$), a number in the industry thought that those levels were persist, rather than believing in mean-reversion.

Certainly, we’ve seen that played out with recent forecasts, where few have predicted that the never-before seen $100/barrel levels were temporary and unsustainable and most predicted prices would remain near or above $100 for the next two decades, as the figure below, from the 2014 DOE Annual Energy Outlook shows.

What most have forgotten is that a decade before, the same forecasters were predicting long-term prices below $30/barrel (roughly $35/barrel in 2014$) as the figure below shows. (My own forecast, labelled SEER, was similarly in error.)  Indeed, when Goldman Sachs suggested oil prices could experience a super-spike to $105/barrel (if supply was disrupted), they were widely censured. Except for a few peak oil advocates, who were not right but lucky, no one was expecting the price levels reached in the past decade, which should have suggested that those prices were unlikely to persist.

Long-term oil prices are much more amenable to quantitative analysis than short-term prices. The lesson that keeps eluding many in the business is an old one: Mark Twain once remarked that a cat which jumps on a hot stove will never jump on a hot stove again, but it won’t jump on a cold one, either. Forecasting and computer models are not the problem, rather bad theories and analysis, poorly constructed models, and sweeping generalizations are the source of error. Avoiding those will get us much closer to better performance, and a future post will describe how to implement such.