AI and efficiency AI and efficiency

Total amount of compute in teraflops/s-days used to train to AlexNet level performance. Lowest compute points at any given time shown in blue, all points measured shown in gray.2, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16

Algorithmic efficiency can be defined as reducing the compute needed to train a specific capability. Efficiency is the primary way we measure algorithmic progress on classic computer science problems like sorting. Efficiency gains on traditional problems like sorting are more straightforward to measure than in ML because they have a clearer measure of task difficulty.A However, we can apply the efficiency lens to machine learning by holding performance constant. Efficiency trends can be compared across domains like DNA sequencing17(10-month doubling), solar energy18(6-year doubling), and transistor density3(2-year doubling).

For our analysis, we primarily leveraged open-source re-implementations19, 20, 21 to measure progress on AlexNet level performance over a long horizon. We saw a similar rate of training efficiency improvement for ResNet-50 level performance on ImageNet (17-month doubling time).7, 16We saw faster rates of improvement over shorter timescales in Translation, Go, and Dota 2:

1. Within translation, the Transformer22surpassed seq2seq23performance on English to French translation on WMT’14 with 61x less training compute 3 years later.

2. We estimate AlphaZero24took 8x less compute to get to AlphaGoZero25level performance 1 year later.

3. OpenAI Five Rerun required 5x less training compute to surpass OpenAI Five26(which beat the world champions,OG⁠(opens in a new window)) 3 months later.

It can be helpful to think of compute in 2012 not being equal to compute in 2019 in a similar way that dollars need to be inflation-adjusted over time. A fixed amount of compute could accomplish more in 2019 than in 2012. One way to think about this is that some types of AI research progress in two stages, similar to the “tick tock” model of development seen in semiconductors; new capabilities (the “tick”) typically require a significant amount of compute expenditure to obtain, then refined versions of those capabilities (the “tock”) become much more efficient to deploy due to process improvements.

Increases in algorithmic efficiency allow researchers to do more experiments of interest in a given amount of time and money. In addition to being a measure of overall progress, algorithmic efficiency gains speed up future AI research in a way that’s somewhat analogous to having more compute.

In addition to efficiency, many other measures shed light on overall algorithmic progress in AI. Training cost in dollars28is related, but less narrowly focused on algorithmic progress because it’s also affected by improvement in the underlying hardware, hardware utilization, and cloud infrastructure. Sample efficiency is key when we’re in a low data regime, which is the case for many tasks of interest. The ability to train models faster29also speeds up research and can be thought of as a measure of the parallelizability30of learning capabilities of interest. We also find increases in inference efficiency in terms of GPU time31, parameters16, and flops meaningful, but mostly as a result of their economic implicationsB rather than their effect on future research progress. Shufflenet13achieved AlexNet-level performance with an 18x inference efficiency increase in 5 years (15-month doubling time), which suggests that training efficiency and inference efficiency might improve at similar rates. The creation of datasets/​environments/​benchmarks is a powerful method of making specific AI capabilities of interest more measurable.

If large scale compute continues to be important to achieving state of the art (SOTA) overall performance in domains like language and games then it’s important to put effort into measuring notable progress achieved with smaller amounts of compute (contributions often made by academic institutions). Models that achieve training efficiency state of the arts on meaningful capabilities are promising candidates for scaling up and potentially achieving overall top performance. Additionally, figuring out the algorithmic efficiency improvements are straightforwardF since they are just a particularly meaningful slice of the learning curves that all experiments generate.

We also think that measuring long run trends in efficiency SOTAs will help paint a quantitative picture of overall algorithmic progress. We observe that hardware and algorithmic efficiency gains are multiplicative and can be on a similar scale over meaningful horizons, which suggests that a good model of AI progress should integrate measures from both.

Our results suggest that for AI tasks with high levels of investment (researcher time and/or compute) algorithmic efficiency might outpace gains from hardware efficiency (Moore’s Law). Moore’s Law was coined in 1965 when integrated circuits had a mere 64 transistors (6 doublings) and naively extrapolating it out predicted personal computers and smartphones (an iPhone 11 has 8.5 billion transistors). If we observe decades of exponential improvement in the algorithmic efficiency of AI, what might it lead to? We’re not sure. That these results make us ask this question is a modest update for us towards a future with powerful AI services and technology.

For all these reasons, we’re going to start tracking efficiency SOTAs publicly. We’ll start with vision and translation efficiency benchmarks (ImageNetG and WMT14), and we’ll consider adding more benchmarks over time. We believe there are efficiency SOTAs on these benchmarks we’re unaware of and encourage the research community tosubmit them here⁠(opens in a new window)(we’ll give credit to original authors and collaborators).

Industry leaders, policymakers, economists, and potential researchers are all trying to better understand AI progress and decide how much attention they should invest and where to direct it. Measurement efforts can help ground such decisions. If you’re interested in this type of work,consider applying⁠to work at OpenAI’s Foresight or Policy team!

Algorithmic Efficiency SOTAs

Submit on GitHub

AlexNet-level performance

79.1% top 5 accuracy on ImageNet

Publication Compute (TFS-Days)Reduction factor Analysis Date

1AlexNet3.1 1 AI and Efficiency Jun 1, 2012

2GoogLeNet0.71 4.3 AI and Efficiency Sep 17, 2014

3MobileNet0.28 11 AI and Efficiency Apr 17, 2017

4ShuffleNet (1x)0.15 21 AI and Efficiency Jul 3, 2017

5ShuffleNet v2 (1x)0.12 25 AI and Efficiency Jun 30, 2018

1EfficientNet (b0)0.069 44EfficientNetMay 28, 2019

ResNet-50-level performance

92.9% top 5 accuracy on ImageNet

1ResNet-5017 1 AI and Efficiency Jan 10, 2015

2EfficientNet (b1)0.75 10EfficientNetMay 28, 2019

Seq2Seq-level performance

34.8 BLEU on WMT-14 EN-FR

1Seq2Seq (Ensemble)465 1AI and ComputeJan 10, 2014

2Transformer (Base)8 61Attention is all you needJan 12, 2017

GNMT-level performance

39.92 BLEU on WMT-14 EN-FR

1GNMT1620 1Attention is all you needJan 26, 2016

2Transformer (Big)181 9Attention is all you needJan 12, 2017

Publication Compute (TFS-Days)Reduction factor Analysis Date

1GNMT1620 1Attention is all you needJan 26, 2016

2Transformer (Big)181 9Attention is all you needJan 12, 2017