Tesla now holds the mantle of Moore’s Law, with the D1 chip introduced last night, so I had to update my favorite graph and see where they lay.
This shift should not be a surprise, as Intel ceded leadership to NVIDIA a decade ago, and a further handoff was inevitable. The computational frontier has shifted across many technology substrates over the past 122 years, most recently from the CPU to the GPU to ASICs optimized for neural networks (the majority of new compute cycles).
Of all of the depictions of Moore’s Law, this is the one I find to be most useful, as it captures what customers actually value — computation per $ spent (note: on a log scale, so a straight line is an exponential; each y-axis tick is 100x).
And what do we see… Humanity’s capacity to compute has compounded for as long as we can measure it, exogenous to the economy, and starting long before Intel co-founder Gordon Moore noticed a refraction of the longer-term trend in the belly of the fledgling semiconductor industry in 1965. Spooky. Like it’s on exponential rails.
In the modern era of accelerating change, it is hard to find even five-year trends with any predictive value, let alone trends that span the centuries. I would go further and assert that this is the most important graph ever conceived (my earlier blog post on its origins and importance: https://flic.kr/p/qS3HU4)
Why the transition within the integrated circuit era? Intel lost to NVIDIA for neural networks because the fine-grained parallel compute architecture of a GPU maps better to the needs of deep learning. There is a poetic beauty to the computational similarity of a processor optimized for graphics processing and the computational needs of a sensory cortex, as commonly seen in neural networks today. A custom chip (like the Tesla D1 ASIC) optimized for neural networks extends that trend to its inevitable future in the digital domain. Further advances are possible in analog in-memory compute, an even closer biomimicry of the human cortex. The best business planning assumption is that Moore’s Law, as depicted here, will continue for the next 20 years as it has for the past 120.
Tesla AI Day video: https://youtu.be/j0z4FweCy4M?t=6340
and summary: https://electrek.co/2021/08/20/tesla-dojo-supercomputer-worlds-new-most-powerful-ai-training-machine/
Leave a Reply