The State of Deep Learning in 2017

In early 2017, ARK published “Deep Learning: An Artificial Intelligence Revolution”. This whitepaper made the case for deep learning as a foundational technology that should transform not only technology companies but every sector of the global economy. Since the paper’s publication, deep learning continues to grow at a rapid pace both in research and in commercialization. In this update, we review four major developments in deep learning that stood out since our initial publication in 2017.

Deep Learning as Software 2.0

The most notable trend was a shift in understanding that deep learning is not just a ‘new tool’: it is a new way to build software. Tesla’s [TSLA] Director of AI, Andrej Karpathy, argues that collecting data and tuning learning systems is much easier than writing code to solve complex problems.

In conventional software development, Software 1.0, programmers spend a significant amount of time prototyping and experimenting with different designs in search of optimal solutions. For problems that are sufficiently complex, like autonomous vehicles, a manual search of the solutions space is not feasible. Deep learning, Software 2.0, automates the laborious search process: simple uniform functions replace complex handcrafted functions, and rigorous iterative search replaces ad hoc human tinkering. The result is a systematic way of building highly complex programs, the best of which exceed human performance in narrow domains.

Software 2.0 is starting to disrupt traditional software verticals which heretofore have benefited from decades of optimization. Google [GOOG] showed that a neural net-based index commonly used in databases outperforms a traditional index by 70% while consuming 90% less memory. Elsewhere, researchers are finding that deep learning produces results that rival the best algorithms focused on the weather and physics simulations that dominate the $25 billion high performance computing market.

In other words, deep learning is eating software. We expect this trend to continue during the next few years as industries and companies wake up to the power of Software 2.0 and discover more ways to apply and capitalize on it.

Algorithmic Improvements

The power of deep learning depends strongly on the design and training of the underlying neural networks. In recent years, neural networks have become complicated, often containing hundreds of layers and requiring millions of data examples for training. In 2017, researchers decided to focus on more efficient neural networks together with reinforcement learning which requires no training data.

While deep learning traditionally has required large amounts of costly or scarce data, a new design – Capsule Networks – could reduce data needs significantly. Capsule Networks group neurons into bundles, helping networks to “learn” objects in 3-dimensions and achieving high performance with fewer examples.

Deep reinforcement learning—a key technology for robotics and self-driving cars—also reached a major milestone in 2017. Updating its AlphaGo program to use reinforcement learning exclusively, DeepMind achieved super-human results in chess, shogi (Japanese chess) and go.[1] Using deep learning and a form of tree search, DeepMind’s artificial intelligence (AI) obsoleted decades of game-specific AI strategies, reinforcing our view that deep learning is broadly applicable and can displace highly optimized conventional software.

Lastly, generative adversarial networks (GANs), made enormous strides in 2017. GANs allow neural networks to generate realistic images from scratch. In their early days, the results were somewhat strange and blurry, as shown on the left below, but did illustrate that deep learning could be useful for content creation in the near future. In 2017 NVIDIA [NVDA] unveiled an improved GAN that creates high resolution images approaching photorealism, as shown on the right below. GANs will extend deep learning’s addressable market into areas such as graphic design, music creation, video games, and drug discovery.

New Deep Learning Chips

Deep learning’s demanding computational requirements have caused an investment boom in new microprocessors, as shown below. In June, industry leader NVIDIA released Volta, the first GPU with dedicated Tensor Cores that accelerate deep learning. Volta can perform 120 trillion operations per second, six times more than its predecessor. Demand has been exceptionally strong, with Amazon [AMZN], Microsoft [MSFT], and various Chinese internet companies adopting it in their data centers. As a result, NVIDIA’s data center revenues doubled to $606 million during the fourth quarter of 2017 based on company filings.[2]

NVIDIA’s rapid growth in deep learning has attracted a horde of competitors, as shown below. ARK has catalogued at least a dozen companies, ranging from industry giants to secretive startups, building deep learning processors. Among them, Google is the most competitive, having released its second generation Tensor Processor Unit (TPU) in May. Although not as powerful individually as the fastest GPUs, TPUs can be linked together to achieve very high performance at relatively low cost. By bringing chip design in-house, Google has morphed into a full stack AI company,[3] giving it a commanding lead in cloud AI services, AI research, and potentially autonomous driving.

We expect to see intense competition among chip vendors in 2018. NVIDIA and Google likely will announce new chip designs at their developer conferences while Intel [INTC] and Graphcore should start sampling their deep learning processors during the first half of the year.

China’s Massive AI Investment

In 2017, the Chinese government woke up to the importance of artificial intelligence. In a policy paper released by the State Council, China stated that it aims to catch up to western countries in AI by 2020, to make major breakthroughs by 2025, and to become the world leader by 2030. It intends to invest billions of dollars in artificial intelligence, creating a $150 billion domestic AI industry over the next decade.[4]

While ambitious government programs should be viewed with healthy skepticism, we believe that China is well positioned to succeed both in AI research and in its commercialization. Among the reasons are the following:

• Data is the fuel for AI. At 700 million, the number of online citizens in China is more than double that of the United States, providing Chinese companies with larger datasets and more monetization opportunities.
• China’s domestic internet giants are sophisticated practitioners of deep learning. Alibaba [BABA], Tencent [TCEHY], Baidu [BIDU], and JD [JD] have trained, deployed, and monetized deep learning products at scale. Their vast customer bases and varied business models provide ample opportunity for growth. Among startups, China’s SenseTime is the world’s most valuable deep learning company with a $2 billion valuation.
• Although Western countries currently lead in deep learning research, China is catching up. Notably, the widely used “ResNet” neural network for image recognition was the work of Microsoft researchers based in Beijing. Baidu also has made important contributions to voice recognition with its “DeepVoice” neural network. In 2016, China published more AI papers than the European Union.
• China’s lax attitude toward privacy makes it easier to collect data and deploy AI services at scale. While Western internet companies are under pressure to limit data collection and governments are restricted by privacy laws, Chinese internet users are accustomed to governments and companies tracking and monitoring their whereabouts, their comments, interests, and so forth. As a result, use cases like mass video surveillance will become much larger markets in China than in western countries.

Deep learning made rapid progress in 2017. In 2018, we will track a number of key milestones including the launch of new deep learning processors and Tesla’s autonomous cross country test drive. We look forward to reviewing progress with you again at the end of the year.

To download and read our full whitepaper, please click here.