Note the pivotal AI moment in 2012 when a neural net run on GPUs crushed the ImageNet competition. Deep Learning and non-CPU architectures blossomed.
From the AI Index 2019: “Prior to 2012, AI results closely tracked Moore’s Law, with compute doubling every two years. Post-2012, compute has been doubling every 3.4 months (for a net increase of ~10,000,000x since then).
The y-axis of the chart shows the total amount of compute, in petaflop/s-days, used to train selected results. A petaflop-day (pf-day) consists of performing 10^15 neural net operations per second for one day, or a total of about 10^20 operations.
In the past year and a half, the time required to train a network on cloud infrastructure has fallen from about three hours in October 2017 to about 88 seconds in July, 2019. During the same period, the cost to train such a system has fallen similarly.”
— p.65 of the AI Index from Stanford HAI (and the OpenAI graph of the recent red datapoints)
Leave a Reply