The extraordinary resilience of U.S. shale oil production continues to confound forecasters, leading to flawed market analysis and undermining OPEC’s strategy to force higher cost producers to shutter output in the lower oil price environment. U.S. independent shale oil producers have adapted to the downturn in oil prices with innovative and more advanced applications of technology to sharply reduce production costs and improve oil field recovery rates to levels unimaginable in the early days of shale development.

When OPEC adopted its strategy to maintain high production levels despite the expected downward pressure on prices in late 2014, shale oil development and production costs were estimated at around $70-$80 per barrel (bbl) compared to the producer group’s much lower costs from $5-$45/bbl. As international oil prices trended lower, more and more of the relatively costlier shale production was expected to be shut-in, leading to a rebalancing of oversupplied markets and a return to higher oil price levels.

When oil prices were trending above $100/bbl, U.S. shale producers were focused on maximizing output at the expense of developing operating efficiencies at the relatively new oil fields. The game changing combination of advanced hydraulic fracturing technology and horizontal well drilling, or fracking, reversed the long decline in U.S. oil production and nearly doubled output from 38-year lows of 5 million barrels per day (mb/d) in 2008 to almost 9.5 mb/d in 2015. Shale, also called tight oil, provided 90 percent of the increase, rising from 500,000 b/d (kb/d) to 4.5 mb/d over the same period.

However, some of the same pioneering companies that developed previously uneconomic shale oil resources rapidly turned their focus to reducing operating per barrel costs and improving efficiencies and productivity as oil prices cascaded lower. While many cash-strapped shale oil producers have fallen victim to lower oil prices, other firms have been able to push new frontiers with more sophisticated applications of technology, drilling techniques, and best management practices used to increase production at lower costs. Production costs have declined on average by 35-40 percent over the past two years, and by more than 50 percent in some producing regions with the best assets such as the prime Permian Basin. Shale oil front runner EOG Resources said in early August it will increase its fracking plans by 30 percent this year, and expects significant profit returns on new wells even as oil has dipped back to $40/bbl. Pioneer Natural Resources recently said that in the Permian Basin the company can compete with Saudi Arabia on costs. Prices for West Texas Intermediate were last trading around $46.50/bbl with other shale producers now following EOG Resources’ lead and increasing activity. Moreover, there has been a feverish run in buying new oil acreage in the coveted Permian Basin in Texas. Shale costs on average now range from $35-50/bbl.

The rapid technological advances in field developments and the vast landscape of shale oil fields, coupled with hundreds of different companies operating the wells, has, not surprisingly, made forecasts of future production levels problematic. The U.S. Energy Information Administration, the primary source of data on tight oil, has consistently made sharp upward revisions to its monthly and annual forecasts for shale oil production. In 2010, the EIA said in its annual energy outlook (AEO) that it did not expect a significant increase in commercial shale oil production until 2023; yet in 2011, output doubled from just under 800 thousand barrels per day (kb/d) to 1.6 mb/d. The EIA said in the 2010 AEO: “With ongoing improvement in oil shale technology, commercial production starts in 2023.” In its 2013 outlook, shale oil production was projected at 2.6 mb/d in 2015, or almost 2 mb/d below actual levels.

The EIA was not alone in underestimating the potential for shale oil growth. The OPEC secretariat forecast in its annual 2012 World Oil Outlook that shale output “is projected to increase from about 1 mb/d in 2012 to 2 mb/d in 2020, before reaching 3 mb/d from 2025 onward.” Another report published by the International Association for Energy Economics in the third quarter of 2013 noted that “U.S. shale oil production would hardly make a dent in the global oil supplies as it would largely offset the decline in U.S. conventional oil production.“

In the EIA’s 2016 AEO released in April, shale oil production in the base case Reference scenario is forecast to decline from 4.89 mb/d in 2015 to 4.27 mb/d in 2016 and 4.19 mb/d in 2017 before edging higher in 2018 on expectations of rising oil prices. The Reference base case projection is a business-as-usual trend estimate reflecting current laws and regulations, known technology, and technological and demographic trends. Under its Reference scenario, the EIA assumes oil prices rise steadily to around $79/bbl in 2020. In August, the EIA issued a special report devoted to shale based on data in the AEO, saying that “From 2015 to 2017, tight oil production is projected to decrease by 700,000 barrels per day in the Reference case, mainly attributed to low oil prices and the resulting cuts in investment. However, production declines will continue to be mitigated by reductions in cost and improvements in drilling techniques.”

The 2016 EIA AEO forecasts production as far out as 2040. Shale oil production is projected to continue its upward trajectory after 2018, breaching the 5 mb/d mark in 2020 and steadily rising to just over 7 mb/d in 2040.

In 2016, the EIA included in its scenarios for tight oil a high oil resource and technology case that allows for the potential role of advanced technologies. Under the high technology scenario, shale output jumps to 4.9 mb/d in 2017, hits a new record of 5.7 mb/d in 2018, and reaches almost 7 mb/d by 2020. By 2027, tight oil output is projected at a lofty 9 mb/d, according to the EIA. By the end of the forecast period in 2040, the EIA projects tight oil output at almost 13 mb/d.

The EIA’s latest annual projections under a high technology scenario are surely a sobering prospect for OPEC. However, given its history of revisions and the multitude of variables that go into the calculations, there is still a fair amount of skepticism about the EIA’s forecasts. That said, the EIA continues to review its data collection process and forecasting methodologies to improve its projections and recently commissioned outside experts to provide more in-depth analysis. The EIA released a study in March it commissioned IHS Markit to conduct on drilling and production costs.

Given the dearth of accurate public information, consulting companies and investment banks are sharpening their own data analysis and forecasts for shale and often have a better track record than the government in predicting future production levels but, of course, that information comes at a price and there are widely varying projections by private companies as well. Even other government agencies are delving into the minutia of data to create their own forecasts for shale given its impact on global oil prices, with recent reports from the U.S Federal Reserve to the U.S. Congressional Budget Office and as far afield as the European Central Bank entering the fray.

Shale Oil and Colonel Drake  

U.S. tight oil production has developed rapidly over the past decade but, somewhat ironically, shale oil was recognized as a potentially valuable U.S. energy resource as early as 1859, the same year Edwin Laurentine Drake, or Colonel Drake, completed his first oil well in Titusville, Pennsylvania, according to a report from the U.S. Department of Energy’s Office of Naval Petroleum and Oil Shale Reserves. More than 150 years ago shale oil products included kerosene and lamp oil, paraffin, fuel oil, lubricating oil and grease, naphtha, and illuminating gas. At the beginning of the 20th century, when the U.S. Navy converted its shipping fleet to run on fuel oil rather than coal, and the country’s economy was transformed by mass production of gasoline-fueled automobiles and diesel-fueled trucks and trains, security of supplies became a critical issue. In 1912, President William Howard Taft, by executive order, established the Naval Petroleum and Oil Shale Reserves (NPOSR) as an emergency source of fuel for the military and the geology of the country’s tight oil resources was mapped in detail.

In 2004, as worries over peak oil gathered pace, the NPOSR produced a two-part report detailing the history and potential of U.S. shale production. The report, “Strategic Significance of America’s Oil Shale Resource,” noted that previous attempts to commercialize shale failed due to lack of technology and low oil price but the government long ago recognized that the economic viability of shale would eventually improve.

More than a hundred years later, U.S. independent oil companies are rapidly building up their knowledge base of the tight oil fields and developing best business practices to efficiently extract this “new” resource in the most cost-efficient methods. In just a few years, these pioneering companies have managed to sharply lower production costs and increase productivity against a backdrop of prices falling by near 70 percent since mid-2014.

Shale oil fields were initially also thought to have a short producing life span given the exceptional rapid and steep decline rates in the first year of production. Just a few years ago, shale fields posted first-year decline rates of 90 percent, which led some analysts to downplay shale’s longer-term role in global oil markets. Conventional oil fields have much lower decline rates at around an average 10 percent. In the wake of lower prices and more experience with tight oil, some producers have managed to improve decline rates to 50 percent.

Improved productivity per rig has also skewed forecasts. Analysts have been closely tracking rigs in operation to gauge future production. When the rig count dropped nearly 80 percent, analysts expected shale output to plummet. However, output defied expectation, falling just 8 percent by July 2016 from a peak of 4.6 mb/d in April 2015. Improved productivity per rig and savvy operators’ decisions to keep drilling, but not complete wells until prices increase – the so-called “drilled but uncompleted” (DUC) wells – have broken the traditional lock-step relationship between rig activity and production as a forecasting tool. The DUCs are considered yet another wildcard that could slow or even reverse declining output.

Injecting more uncertainty into forecasts, the major oil companies are now expanding into the development of tight oil, which until recently was largely considered the exclusive purview of U.S. independents. Companies like BP, Shell, ExxonMobil, and Chevron, which typically develop the multibillion-dollar projects that are now cost prohibitive in the lower oil price environment, are turning their hand to tight oil projects. So far, with the exception of ExxonMobil, international oil companies have had limited success compared to the independent producers. Tight oil wells drilled by the majors were on average one-third less productive than the top 10 operators, according to data analytics firm NavPort. But like the independents, the majors will, in all likelihood, continue to gain experience in extracting tight oil from the complex, low permeability, and deep geological formations, throwing another wrench into forecasting shale oil production trends.

The fast-evolving advanced technology and innovative approaches to tight oil development and production by cutting-edge companies will continue to challenge forecasters to build their knowledge base over the next few years. For OPEC, the massive cuts in capital expenditures for the megaprojects and declining non-OPEC production hold the key to its market share strategy, albeit as long as oil prices stay at lower levels for longer. Oil markets are historically cyclical and lower capital expenditure for conventional oil projects today will eventually lead to tighter supplies in the future, but the timing of a recovery to a more balanced market will remain a moving target as long as fast-cycle, technology-driven shale oil production continues to exceed expectations.