“We’ll soon see the power of computing increase way more than any other period. There will be trillions of products with tiny neural networks inside.”
Intel has long ago ceded leadership for Moore’s Law. And so, understandably, they have trumpeted the end of Moore’s Law for many years. To me, it sounds a lot like Larry Ellison’s OpEd declaring the end of innovation in enterprise software, just before cloud computing and SaaS took off. In both cases, the giants missed the organic innovation bubbling up all around them.
For the past seven years, it has not been Intel but NVIDIA that has pushed the frontier of Moore’s processor performance/price curve. For a 2016 data point, consider the NVIDIA Titan GTX. It offers 10^13 FLOPS per $1K (11 trillion calculations per second for $1,200 list price), and is the workhorse for deep learning and scientific supercomputing today. And they are sampling much more powerful systems that should be shipping soon. The fine-grained parallel compute architecture of a GPU maps better to the needs of deep learning than a CPU. There is a poetic beauty to the computational similarity of a processor optimized for graphics processing and the computational needs of a sensory cortex, as commonly seen in neural networks today.
Here are some of Huang’s provocative prognostications from WSJ.D Live (and some photos I took of him):
It turns out we created an AI company. We power deep learning algorithms in the car. To drive autonomously, the car needs to do perception, reasoning, planning, and learning. These are a big part of AI.
Deep Learning has taken perception to a level that is superhuman. We have eyes around the car. We are never intoxicated or angry. It can look around corners and see things you can’t see, helping you even when you are driving.
To drive autonomously, we have to predict where everything will be in near future. We do path planning and detect objects. But we need to invert the logic. When humans drive, we don’t keep checking “there is no tree in the way; there is no boat in the way” and so on. We need to train on what is safe, not on all the things to avoid.
Elon is right. The AI gets better and better over use. It needs road miles.
We want to turn the car into an AI itself. I want to talk to it and have it respond with a sultry voice. You can ask to make a call for you from calendar.
The autonomous car improves its driving over time, whereas the human capability decreases over time as we age.
The AI should get a drivers license.
Computing will increase at product of Moore’s Law and Metcalfe’s Law. We’ll soon see the power of computing increase way more than any other period.
AI takes a different process, a new software approach, new tools, a new computational architecture. Microsoft just announced Cognitive Studio / CNTK yesterday.
To enable AI, we will shift deep learning from CPUs to a new computing architecture with GPUs.
We will embed inference engines into everything: little robots, sensors, into the factory, the machines building the machines. There will be trillions of products with tiny neural networks inside.
Every search query that uses Hadoop today will move to deep learning. Every query will invoke a billion calculations when we add AI to our apps.
Leave a Reply