Study explores how there may be ways to make AI more energy efficient by starting smaller

New research investigates why curriculum learning struggles in large neural networks and suggests paths to improved energy efficiency.

The rapid rise of artificial intelligence (AI) has reshaped industries and research fields, but its growing demand for computational power raises significant concerns. Neural networks, the backbone of many AI systems, are notoriously resource-intensive, particularly during their training phases. A recent study, published in the Journal of Statistical Mechanics: Theory and Experiment (JSTAT), explores the relationship between the size of neural networks and the effectiveness of curriculum learning.

Led by Luca Saglietti at Bocconi University, the research investigates why overparameterised neural networks—those with an excessive number of parameters—tend to underutilise curriculum learning. While curriculum learning, inspired by human educational models, has shown promise in improving resource efficiency for smaller networks, the findings suggest that its impact diminishes as network size grows.

“What we’ve seen is that an overparameterised neural network doesn’t need this path because, instead of being guided through learning by examples, it’s guided by the fact that it has so many parameters—resources that are already close to what it needs,” explains Saglietti.

The Role of Overparameterisation

Overparameterisation is a widely used strategy in machine learning, involving the deployment of large networks during the training phase to enhance task precision. These networks, often containing millions or even billions of parameters, can achieve remarkable accuracy in tasks such as image recognition or natural language processing. However, this comes at a steep cost in energy consumption and environmental impact.

The study draws parallels between overparameterisation and the Lottery Ticket Hypothesis, a concept suggesting that large networks contain smaller, efficient subnetworks capable of performing the same tasks. It is called the Lottery Ticket Hypothesis because finding the optimal subnetwork within a large neural network is likened to holding a winning lottery ticket among many possible configurations. During training, these subnetworks emerge as the model “learns,” allowing the overall network size to be reduced afterward. However, if training begins with a smaller network, the performance often suffers.

The researchers explored whether curriculum learning—providing data in a structured order of increasing complexity—could mitigate the need for overparameterisation. While promising in smaller networks, the approach appears less relevant for larger systems, which already possess the capacity to process complex data without requiring structured input.

Insights from Curriculum Learning

Curriculum learning mimics the gradual progression of human learning. For instance, a novice pianist begins with simple tunes like “Twinkle Twinkle Little Star” before attempting advanced compositions by Chopin. This principle, when applied to neural networks, involves feeding data in a curated order to optimise training efficiency.

Surprisingly, the study found that in overparameterised networks, the order of input data had little impact on performance. Instead, these large systems leveraged their vast resources to learn effectively, regardless of the data’s organisation.

Despite this, the research offers a hopeful perspective. By starting with smaller networks and applying curriculum learning, the team observed performance improvements over randomised input training. This suggests that the strategy could still play a role in developing more resource-efficient models, particularly when paired with smaller initial network sizes.

Saglietti elaborates, “At least within the experiments we conducted, we observed that if we start with smaller networks, the effect of the curriculum—showing examples in a curated order—begins to show improvement in performance compared to when the input is provided randomly. This improvement is greater than when you keep increasing the parameters to the point where the order of the input no longer matters.”

Towards Sustainable AI

The findings come at a pivotal moment for AI development. As systems like ChatGPT and other advanced machine learning models become ubiquitous in scientific and industrial applications, their energy demands continue to rise. Training these systems often requires massive servers and vast computing resources, with significant environmental implications.

One of the study’s key contributions is its potential to inform strategies for reducing the environmental footprint of AI. By optimising the training process, researchers could achieve similar levels of performance with fewer resources. This aligns with growing efforts to make AI development more sustainable, addressing concerns over energy consumption and emissions.

“Our results suggest that if we start with smaller networks, the effect of the curriculum—showing examples in a curated order—begins to show improvement,” Saglietti emphasises. This approach could offer a pathway to balance efficiency and performance, enabling AI systems to do more with less.

Implications for Research and Industry

The study’s findings have far-reaching implications for both research and industry. For academic researchers, understanding the dynamics of overparameterisation and curriculum learning could lead to more efficient experimental designs and resource usage. For industry, particularly in fields reliant on large-scale AI models, the potential cost savings from optimised training processes could be substantial.

Moreover, the insights contribute to a broader conversation about the future of AI. As the technology continues to expand its capabilities, developing models that minimise energy consumption while maintaining performance will be critical. The work of Saglietti and colleagues provides a foundation for these efforts, combining theoretical exploration with practical applications.

While overparameterisation remains a cornerstone of high-performance AI, this study underscores the importance of revisiting traditional training strategies. By refining approaches like curriculum learning and rethinking network design, researchers can address some of the most pressing challenges in AI development.