American university develops "early bird method" to train deep neural networks, which can reduce energy consumption by 10.7 times

Rice University has developed an energy-efficient method for training deep neural networks (DNNs), a form of AI (artificial intelligence) behind self-driving cars, smart assistants, facial recognition and a variety of high-tech applications, according to foreign media reports.

(Image source: Rice University)

Researchers at Rice University and Texas A&M University said that Early Bird can reduce the energy consumed when training DNNs by 10.7 times, and the accuracy level is the same as or even better than conventional training methods.

"Recent breakthroughs in AI have been driven primarily by the introduction of larger and more expensive DNNs," the researchers said. "But training such DNNs requires a lot of energy. To bring more innovative products to market, we must find greener training methods that both address environmental issues and reduce financial barriers to AI research."

Training cutting-edge DNNs is expensive, and getting more expensive. A 2019 study found that the computational requirements for training state-of-the-art deep neural networks increased 300,000 times between 2012 and 2018. Another study also showed that the energy consumption of training an elite DNN is equivalent to the carbon dioxide emissions of 5 American SUVs over their lifetime.

DNNs contain millions or even billions of artificial neurons that learn to perform specific tasks. Without any explicit programming, deep networks of artificial neurons can learn to make human-like decisions, or even surpass human experts, by "learning" from a large number of previous examples. For example, if a DNN studies pictures of cats and dogs, it will learn to recognize cats and dogs. In 2015, AlphaGo, a deep neural network trained to play chess, successfully defeated a professional chess player after learning from tens of thousands of previous chess games.

"The current state-of-the-art approach to DNN training is called progressive pruning and training," the researchers said. "First, you need to train a dense, large neural network, then remove seemingly unimportant parts, like pruning a tree. Then, you retrain the pruned network to restore its performance, as performance degrades after pruning. In practice, you need to prune and retrain multiple times to get good performance. The first step, training a dense, large network, is the most expensive, so you need to get the final, fully functional, pruned network in the first step, the 'early-bird ticket'."

By looking for key network connection patterns in the early stages of training, the researchers discovered the existence of "Early Bird Tickets" and used them to simplify DNN training. In experiments on various DNN models on benchmark datasets, the researchers found that the probability of "Early Bird" appearing in the initial training stage was only one-tenth, or even less.

"Our method can automatically identify early bird tickets 10% or more earlier, before training dense and large networks," the researchers said. "This means that compared with traditional methods of training DNNs, it can reduce the time required to train DNNs by about 10% or more, and can achieve the same or even higher accuracy, thus saving both computation and energy consumption."